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CHAPTER  I 
THE  PROBLEM 

I.  NATURE  AND  SCOPE 

The  problem  stated.  The  purpose  of  this  study  is  to  de- 
termine if  there  are  certain  external  characteristics  which 
distinguish  multiple-choice  items  of  high  validity  from  those 
of  low  validity  and  to  explore  the  possibility  of  formulating 
rules  for  item  preparation  which  will  insure  the  incorporation 
of  these  characteristics. 

The  scope  of  the  study.  The  investigation  will  be  con- 
cerned for  the  most  part  with  items  containing  five  responses 
which  are  designed  for  use  in  achievement  tests.  Items  of  the 
type  prepared  for  attitude  questionnaires,  interest  invento- 
ries, and  similar  multiple-response  instruments  are  not  in- 
cluded. Because  of  the  popular  use  of  the  four-choice  item  in 
instructor-made  tests  the  value  of  the  findings  in  this  study 
will  be  tested  for  the  four-choice  as  well  as  the  five-choice 
item. 

II.  DEFINITION  OF  TERMS 

General  terms  with  limited  meanings.  Terms  which  appear 
in  the  statement  of  the  problem  are  defined  in  the  sense  in 
which  they  will  be  used  throughout  the  study. 

Multiple-choice  item.  A test  item  which  consists  of  a 
question  followed  by  four  or  five  suggested  answers  only  one  of 
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which  is  correct,  or  an  incomplete  statement  followed  by  four  or 
five  suggested  conclusions  only  one  of  which  is  correct. 

Outward  characteristics . The  easily  discernible  and  ob- 
jectively determined  features  which  are  the  result  of  the 
editorial  preparation  of  an  item  such  as  the  positive  or  nega- 
tive nature  of  the  statement,  the  clarity  of  expression,  and 
the  relation  of  choices  to  each  other  in  respect  to  length, 
form,  and  meaning. 

Validity . - The  value  of  an  item  as  a measure  of  achieve- 
ment in  a given  subject-matter  field  as  determined  by  the 
degree  to  which  it  discriminates  between  achievers  and  non- 
achievers in  that  field. 

III.  JUSTIFICATION 

Trial  and  error  method  slow  and  unwieldy.  The  construc- 
tion of  valid  multiple-choice  items  in  building  achievement 
tests  has  always  been  a laborious  and  time-consuming  task.  No 
objectively  established  procedures  have  been  available  for  the 
writer  of  such  items  to  follow.  Whatever  has  existed  in  the 
way  of  suggestions  and  guides  to  item-writing  has  been  of  the 
rule-of-thumb  variety.  Common  sense  and  the  rules  of  grammar 
have  provided  the  only  absolute  guides  as  items  were  written, 
administered  in  tests,  and  their  validity  determined.  Items 
of  low  validity  are  either  revised  or  excluded  and  replaced 

with  a new  item  covering  the  same  material.  These  revised  and 
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new  items  are  in  turn  analyzed  end  the  revision  continued 
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until  tests  consisting  of  setisf ac torily  discriminating  items 
are  the  result. 

A challenge  to  test  makers.  This  slow,  cumbersome,  and 
expensive  method  of  developing  items  presents  a challenge  to 
test  makers.  It  is  doubtful  that  an  item  can  ever  be  prepared 
to  which,  prior  to  its  use,  a coefficient  of  validity  and  an 
index  of  ease  can  be  given  which  will  stand  the  test  of  use. 
But  it  seems  probable,  however,  that  certain  criteria  might  be 
determined  which  distinguish  valid  items  from  poor  items.  If 
the  distinguishing  characteristics  of  a good  item  can  be  dis- 
covered, better  rules  for  writing  such  items  can  be  prepared 
than  the  loose  and  unproved  "suggestions"  now  followed,  while 
the  necessity  for  experimentation  and  revision  can  in  all  pro- 
bability never  be  entirely  eliminated,  a more  definite  know- 
ledge of  the  characteristics  of  a good  item  should  make  it 
possible  to  write  items  that  require  less  and  less  of  such 
treatment . 

IV.  REVIEW  OP  PREVIOUS  RESEARCH 

No  comprehensive  study  of  the  sub.ject.  A search  of  the 

literature  of  the  past  twenty  years  in  the  field  of  objective 

testing  reveals  no  comprehensive  treatment  of  multiple-choice 

item  preparation.  Many  studies  have  been  made  on  the  question 

of  item  validity  from  the  statistical  standpoint.  There  are 

available  reports  of  "pin-point"  studies  which  are  concerned 
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with  a single  characteristic  of  an  item.  There  are  those 
writers  who  treat  the  problem  subjectively  by  stating  how 
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items  should  be  written  but  provide  no  objective  proof  of  the 
value  or  necessity  of  their  suggestions. 

Studies  concerning  item  validity.  Among  the  writers  on 
item  validity  Flanagan-*-  has  made  several  contributions  in 
connection  with  the  work  of  the  Cooperative  Test  Service  of 
the  American  Council  on  Education  and  the  development  of  the 
National  Teacher  Examinations.  Guilford* 2  has  developed  a 
technique  for  the  validation  of  test  items.  Kelley^  also 
has  investigated  the  problem  of  validity.  All  of  these  studies 
are  concerned  with  empirically  determined  statistical  validity. 
Remmers^  and  his  associates  have  investigated  widely  in  the 
field  of  reliability  of  multiple-choice  tests.  There  is  no 


^•Flanagan,  John  C.  "Proposed  Procedure  for  Increasing  the 
Efficiency  of  Objective  Tests",  Journal  of  Educational 
Psychology,  Vol.  28,  pp . 17-21,  January,  1937 

"Preliminary  Study  of  the  Validity  of  the 

1940  Edition  of  the  National  Teacher  Examinations",  School 
and  Society,  Vol.  54,  pp.  59-64,  July  26,  1941 

2 Guilford,  J.  P.  "The  Phi  Coefficient  and  Chi  Square  as 
Indices  of  Item  Validity",  Psychometric a,  Vol.  6,  pp . 11-19, 
February,  1941 

^Kelley,  T.  L.  "The  Selection  of  Upper  and  Lower  Groups 
for  the  Validation  of  Test  Items",  Journal  of  Educational 
Psychology,  Vol.  30,  pp.  17-24,  January,  1939 

^Remmers,  H.  H.  and  Adkins,  R.  M.  "Reliability  of  Multiple- 
Choice  Measuring  Instruments,  a Function  of  the  Spearman-Brown 
Prophecy  Formula",  Journal  of  Educational  Psychology,  Vol.  33, 
pp.  385-390,  May,  1942 

(This  is  one  of  many  articles  of  which  Remmers  is 
joint-author  reporting  investigations  in  various 
sub ject-lnatter  fields.) 
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mention  of  any  attempt  to  associate  either  validity  or  relia- 
bility of  a multiole-choice  item  with  its  content  and  form. 

"Pin-point"  studies.  There  are  several  studies  which 
deal  with  a single  characteristic  of  a multiple-choice  item. 
Matthews-*-  and  Symonds^  both  have  investigated  the  order  of 
choices.  McNamara  and  Weitzman^  have  reported  the  results  of 
their  work  in  the  Navy  testing  program  in  connection  with  the 
order  of  choices  and  the  nature  of  the  incorrect  responses. 
Ruch^  and  his  associates  were  early  investigators  in  the  field 
of  multiple-choice  tests  and  stressed  the  importance  of 
"DO-NOT-GUESS"  instructions  to  reduce  the  effect  of  chance  on 


Matthews,  C.  0.  "The  Effect  of  the  Order  of  Printed  Res- 
ponse Words  on  an  Interest  Questionnaire",  Journal  of 
Educational  Psychology.  Vol.  20,  pp.  128-134,  February,  1929 

g 

Svmonds,  P.  M.  "Influence  of  Order  of  Presentation  of 
Items  in  Ranking",  Journal  of  Educational  Psychology,  Vol.  27, 
pp . 445-449,  September,  1936 

^McNamara,  W.  J.  and  Weitzman,  E.  "Effect  of  Choice  Place- 
ment on  the  Difficulty  of  Multiple-Choice  Questions",  Journal 
of  Educational  Psychology,  Vol.  36,  pp.  103-113,  February, 1945 

"Apt  Hse  of  the  Inept  Choice  in  Multiple- 

Choice  Testings",  Journal  of  Educational  Research.  Vol.  39, 
pp.  517-522,  March,  1946 

4Ruch,  G.  M.  and  Foster,  R.  R.  "On  Corrections  for  Chance 
in  Multiple-Response  Tests",  Journal  of  Educational  Psychology, 
Vol.  18,  pp.  48-51,  January,  1927 

and  De  Graff,  M.  H.  "Corrections  for  Chance 

and  ’Guess’  vs.  ’Do  not  Guess'  Instructions  in  Multiple- 

Response  Tests",  Journal  of  Educational  Psychology.  Vol.  17, 

pp . 368-375,  September,  1926 
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total  scores.  Votaw,1  on  the  other  hand,  in  his  study  of  the 
effect  on  validity  of  the  manner  of  response,  emphasized  the 
importance  of  requesting  response  to  all  items.  Stalnaker  and 
Stalnaker,^  as  well  as  Kelley, 3 report  experiments  in  selecting 
distracters.  Wesman^  reports  the  effect  on  validity  of  un- 
answered items  and  the  use  of  "none  of  these"  as  a possible 
choice.  This  list  indicates  the  nature  of  the  work  that  has 
been  done  but  does  not  include  all  of  the  many  studies  which 
are  concerned  with  some  phase  of  item  validity  or  with  some 
characteristics  of  the  individual  item. 


^-Votaw,  D.  F.  "Notes  on  Validation  of  Test  Items  by  Com- 
parison of  Widely  Snaced  Croups",  Journal  of  Educational 
Psychology,  Vol.  25,  pp.  185-191,  March,  1934 

"Effect  of  DO-NOT-GUESS  Directions  upon  the 

Validity  of  True-False  or  Multiple-Choice  Tests",  Journal  of 
Educational  Psychology.  Vol.  27,  pp.  698-703,  December,  1936 

and  Danforth,  L.  "Effect  of  the  Method  of 

Response  upon  the  Validity  of  Multiple-Choice  Tests",  Journal 
of  Educational  Psychology,  Vol.  30,  pp . 624-627,  March,  1939 

p 

Stalnaker,  J.  M.  and  Ruth  C.  "Chance  vs.  Selected  Distrac- 
ters in  a Vocabulary  Test",  Journal  of  Educational  Psychology. 
Vol.  26,  pp.  161-168,  February,  1935 

^Kelley,  V.  H.  "An  Experience  with  Multiple-Choice  Vocabu- 
lary Tests  Constructed  by  Two  Different  Procedures",  Journal 
of  Experimental  Education,  Vol.  5,  pp . 249-250,  March,  1937 

4 

Wesman,  A.  G.  Active  vs.  Blank  Resoonses  to  Multiple- 
Choice  Items",  Journal  of  Educational  Psychology,  Vol.  38, 
pp.  89-95,  February,  1947 

and  Bennett,  G.  K.  "Use  of  ’None  of  these’  as 

an  Option  in  Test  Construction",  Journal  of  Educational 
Psychology,  Vol. '37,  pn . 541-549,  December,  1946 
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General  discussion  of  item  preparation.  Many  text  books 


on  objective  testing  discuss  the  preparation  of  multiple-choice 
items.  Typical  of  recent  publications  of  this  type  is  that  by 
Remmers  and  Gage.-*-  The  suggestions  which  they  present  are  for 
the  most  part  offered  without  supporting  proof  of  their  value. 
Thev  are  mentioned  here  because  they  represent  summaries  of 
current  thinking  on  the  question  of  item  preparation  and  be- 
cause they  will  be  referred  to  in  detail  in  the  following 
chapter. 

^Remmers,  H.  H.  and  Gage,  N.  L.  Educational  Measurement  and 
Evaluation,  Harper  and  Brothers,  New  York,  1945,  pp.  168-171 
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CHAPTER  II 

PRESENT  DAY  GUIDES  FOR  ITEM  WRITING 

I.  A SUMMARY  OF  CURRENT  THOUGHT 
Purpose  of  the  summary.  It  is  the  purpose  of  this  chapter 
to  bring  together  in  a definite  and  concise  form  the  various 
suggestions  and  findings  contained  in  the  contributions  of  the 
writers  already  mentioned  as  well  as  in  the  reported  work  of 
manv  others.  In  order  to  nresent  this  summary  in  a readily 
usable  form,  each  point  will  be  stated  in  the  form  of  a ques- 
tion. A check  list  for  possible  use  in  item  writing  and  in 
the  evaluation  of  items  already  written  will  consequently  be 
evolved. 

The  item  and  its  Darts . In  the  preparation  of  this  check 
list  an  outline  for  item  preparation  suggested  by  Mosier  and 
others-*-  has  been  followed  to  a great  extent.  The  development 
of  the  item  has  been  divided  into  four  steps  each  of  which  has 
been  given  thorough  consideration.  These  four  steps  are  con- 
cerned with  (1)  the  item  as  a whole,  (2)  the  premise,  (3)  the 
correct  answer,  end  (4)  the  distracters.  The  remainder  of  the 
chapter  will  be  devoted  to  a discussion  of  the  questions  in- 
cluded in  the  check  list. 

%osier,  Charles  I.,  Myers,  M.  Claire,  and  Price,  Helen  G. 
"Suggestions  for  'the  Construction  of  Multiple-Choice  Test 
Items",  Educational  and  Psychological  Measurement,  Vol.  5,  No. 3 
pp . 261-271,  Autumn  1945 
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II.  A CHECK  LIST  FOR  ITEM  EVALUATION 
The  Item  as  a whole.  An  analysis  of  the  material  which  is 
concerned  with  the  item  as  a whole  shows  that  the  following 
points  may  be  of  importance  in  this  area: 

1 . Does  the  premise,  combined  with  each  choice,  consti- 
tute a complete  unit  of  thought  ideologically?  To  be 
meaningful  it  is  obvious  that  an  item  must  pose  a 
question  and  present  logical  choices  in  a manner  which 
can  be  comprehended  by  the  examinee.  If  choices  are 
not  related  to  the  premise  ideologically  they  ere 
obviously  incorrect. 1 


If  your  aircraft  is  already  equipped  with  a 
magnetic  compass,  sensitive  altimeter,  turn 
and  bank,  and  airspeed  indicator,  which  of 
the  following  is  most  essential  to  dead  reck- 
oning navigation? 

a.  Rate  of  climb. 

b.  Thermometer. 

c.  Driftmeter. 

d.  Clock. 

e.  Cryro  compass. 


Figure  1. 

An  Ideologically  Unified  Item 


In  the  above  item  (Figure  1)  the  premise  lists  a 


■^Lang,  Albert  R.  Modern  Methods  in  Written  Examinations. 
Houghton  Miff lin' Company,  Boston,  1930,  p.  138 
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number  of  instruments  to  which  must  be  added  one  of 
the  five  choices.  All  of  the  four  alternates  as  well 
as  the  correct  choice,  "clock”,  are  instruments  found 
in  the  cockpit  of  an  airplane.  Each  of  them,  ideologi- 
cally, mieht  constitute  a complete  unit  of  thought  with 
the  premise. 

2.  Does  the  premise,  combined  with  each  choice,  constitute 
a complete  unit  of  thought  grammatically?  The  con- 
fusion which  might  result  from  poor  grammatical  con- 
struction in  an  item  is  emphasized  by  Rinsland.^-  The 
item  illustrated  in  Figure  2,  taken  from  a test  de- 
signed for  college  freshmen,  shows  how  an  item  whose 
parts  are  disjointed  calls  for  more  than  knowledge  of 
the  subject  to  determine  the  correct  answer. 

Which  of  the  following  expressed  the  thesis  of 

Ripley  in  The  Races  of  Europe? 

a.  It  reinforced  the  Nordic  myth. 

b.  It  refuted  the  arguments  of  Gobineau. 

c.  It  demolished  the  theory  of  an  Aryan  race. 

d.  It  emphasized  the  historical  reasons  for 
anti-Semitism. 

Figure  2. 

A Grammatically  Disjointed  Item 


^Rlnsland,  Henry  D.  Constructing  Tests  and  Grading  in 
Elementary  and  High  School  Subjects,  Prentiss-Hall,  New  York, 
1937,  p.  19 
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3 . Does  the  premise,  combined  with  the  right  answer. 


constitute  a definite  and  true  concept?  It  is  of 
utmost  importance  that  the  item  be  so  constructed  that 
the  premise  together  with  the  correct  answer  becomes  a 
statement  the  truth  and  accuracy  of  which  can  be  estab- 
lished beyond  controversy Common  mistakes  of  this 
nature  are  false  generalizations,  mistaking  theory  for 
proven  fact,  lack  of  scientific  classification  and/or 
definition,  and  errors  of  tradition. ^ In  a preceding 
illustration  (Figure  1)  the  correctness  of  the  correct 
answer  in  connection  with  the  provisions  of  the  pre- 
mise is  accepted  without  question. 

4 . Is  the  idea  stated  clearly  and  directly  with  the  answer 
an  important  part  of  the  statement?  An  item  is  good 
only  when  it  presents  to  the  examinee  the  question 
which  was  in  the  mind  of  the  examiner.  If  it  is  worded 
in  a loose,  cumbersome,  or  ambiguous  manner  confusion 
results. ^ Only  those  terms  concerning  which  there  can 
be  no  cause  for  misunderstanding^  should  be  used. 

^Lefever,  D.  Weltey  "Dangers  and  Values  in  Teecher-Mede 
Tests",  Education,  Vol.  53,  pp . 409-413,  March,  1933 

^Diamond,  L.  N.  "Testing  the  Test  Makers",  School  Science 
and  Mathematics,  Vol.  32,  pp . 490-502,  May,  1932 

^Foley,  L.  "Test  of  Tests",  School  Review,  Vol.  51, 
pp.  360-368,  June,  1943 

4 ' 

Eskridge,  T.  J.  "Factor  of  Verbalization  - a Further  Limi- 
tation of  the  Multiple-Choice  Test",  Journal  of  Educational 
Research,  Vol.  35,  pp . 326-333,  January,  1942 
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5 . Does  the  Item  call  for  problem  solving,  interpretation 
of  dat a,  or  for  the  recall  of  factual  information? 

The  validity  of  an  item  may  be  definitely  affected  by 
the  degree  to  which  it  is  thought-provoking  in  its 
nature.  One  writer^  holds  that  something;  more  should 
be  asked  for  in  an  item  than  recall  of  memorized 
facts.  The  item  in  the  criterion  grouos  are  of  three 
general  types,  problem  solving  (Figure  3),  interpre- 
tation of  data  (Figure  4),  and  recall  of  factual 
information  (Figure  5). 


If  3 in.  on  a map  represents  a distance  of 
1000  miles,  a distance  of  450  miles  would 
be  represented  on  a map  by: 

a.  2 . in. 

b.  2.5  in. 

c.  1.35  in. 

d . 5 . in . 

e.  1.25  in. 

Figure  3. 

A Problem  Solving  Item 


Magill,  W.  H.  "The  Influence  of  the  Form  of  Item  on 
the  Validity  of  Achievement  Tests",  Journal  of  Educational 
Psychology , Vol.  25,  pp . 21-28,  January,  1934 
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Prom  NA  to  DY  flight  at  15,0UU  ft.  (the 
assigned  altitude)  will  be  made: 

a.  Above  a cirrostratus  overcast. 

b.  Through  a region  of  light  to 
moderate  icing. 

c.  In  moderate  to  heavy  turbulence. 

d.  Above  an  altostratus  overcast. 

e.  Around  scattered  thunderstorms. 

Pi sure  4. 

An  "Interpretation  of  Data"  Item 


You  are  preparing  for  an  examination  and 
wish  to  review  some  facts  about  night 
vision.  You  should  consult: 

a.  Radio  Data  and  Plight  Information. 

b.  Radio  Pacility  Charts. 

c.  PIP. 

d.  TM  1-205  Air  Navigation. 

e.  Air  Traffic  Rules. 

Figure  5. 

A "Recall  of  Factual  Information"  Item 

6 . Is  the  item  as  direct  a measure  as  possible  of  the 

points  of  mental  attainment  under  consideration?  In 

preparing  multiple-choice  items  it  is  considered 

essential  by  some^  that  the  item  be  a definite  test  of 

the  facts  and  skills  involved  in  the  material 
✓ 

% a gill,  W.  H.  Loc.  cit. 
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under  examination.  If  the  item  covers  a point  in 
mathematical  skill,  it  should  be  so  devised  that  the 
correct  answer  can  be  identified  only  by  performing 
the  necessary  mathematical  computation  correctly  as  in 
Figure  3.  If  it  is  an  item  dealing  with  interpreta- 
tion of  a weather  map,  precise  questions  should  be 
asked  concerning  data  shown  on  a map  (Figure  5). 

What  is  the  position  of  the  correct  choice?  On  this 
question  there  seems  to  be  some  difference  between  the 
subjective  opinions  of  the  "arm-chair " test  construc- 
tors and  the  objective  findings  of  the  few  who  have 
experimented  in  the  field.  Greene  and  others-1-  suggest 
that  random  occurrence  of  correct  choices  should  be 
employed.  McNamara  and  Weitzman,^  however,  found  that 
in  five-choice  items  the  fourth  position  had  the 
greatest  difficulty  level  as  shown  by  the  following 
figures : 

Position  Difficulty  Level 


1 

2 

3 

4 

5 


77 

76.5 

76.4 

74.7 

75.9 


•^-Greene,  Harry  A.,  Jorgensen,  Albert  N.,  and  Gerberich,  J. 
Raymond  Measurement  and  Evaluation  in  the  Secondary  School. 
Longmans,  Green  and  Company,  New  York,  1943,  pp.  193-194 

^McNamara,  W.  J.  and  Weitzman,  E.  Loc . cit. 
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In  a very  early  study,  Matthews-*-  found  that  there  was 
a marked  tendency  to  overrank  choices  coming  early  on 
the  list  and  Symonds^  says  that  this  tendency  to 
overrank  the  early  and  to  underrank  the  later  choices 
is  directly  proportional  to  the  distance  of  the  choice 
from  the  middle  term.  In  view  of  these  conflicting 
ideas  and  facts,  the  question  still  remains  open. 

8.  What  is  the  Index  of  Ease?  Lack  of  difficulty  in  an 
item  is  obviously  a cause  of  low  validity. ^ The 
easier  it  is  the  less  discriminating  it  becomes.  In 
constructing  achievement  tests  it  seems  that  this 
point  may  be  of  extreme  importance  because  of  the  fact 
that  some  essential  points,  necessarily  covered  by 
items,  will  have  been  learned  by  all.  Just  how  many 
of  these  can  be  included  and  still  leave  the  test  as  a 
whole  a valid  instrument  raises  a very  interesting 
question. 

9 . Is  the  item  free  from  grammatical,  spelling,  and 
typographical  errors?  This  question  is  not  to  be  con- 
strued as  overlapping  the  coverage  of  Question  1. 

This  refers  to  incidental  errors  in  the  item  as  a whole 


^Matthews,  C 

. 0. 

Loc  . cit. 

^Symonds,  P. 

M. 

Loc . cit. 

'Z. 

Diamond,  L. 

1J. 

Loc.  cit. 
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and  not  to  the  grammatical  relationship  between  pre- 
mise and  conclusion.  All  writers  on  the  subject  are 
agreed  on  the  necessity  for  grammatical  correctness. 

10 . Are  answers  arranged  horizontally  or  vertically?  At 
first  glance  it  appears  that  the  arrangement  of 
answers  must  necessarily  be  governed  by  their  length. 

In  the  conventional  printed  column  it  is  obviously 
impossible  to  arrange  five  answers  horizontally  con- 
taining; more  than  one  word.  However,  Matthews^  main- 
tains that  the  effect  of  order  is  much  greater  when 
items  are  presented  vertically  than  when  presented 
horizontally. 

11 . Is  the  item  self-contained  or  does  it  require  the  use 
of  reference  material  not  included  in  the  test?  Thi s 
question  was  suggested  by  the  test  items  under  con- 
sideration in  this  study.  Logarithm  tables,  aeronau- 
tical charts,  etc.  were  supplied  for  use  with  some  of 
the  tests.  The  effect  of  the  required  use  of  these 
materials  on  item  validity  is  considered  worthy  of 
investigation . 

12 . How  many  words  are  used  in  stating  the  item  as  a whole? 
The  question  of  the  wordiness  of  an  item  presents  it- 
self as  a possible  contributing  cause  of  the  difference 

in  validity  among  items. 

✓ 

•^Matthews,  C.  0.  Loc . cit. 
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The  premise.  The  following  questions  are  suggested  by  an 
examination  of  the  suggestions  found  in  the  literature  on  the 
sub j ec t . 

1 . Is  a definite  task  clearly  and  unambiguously  set? 

Ambiguous,  equivocal,  and  involved  statements  as  well 
as  broad  general  statements  should  be  avoided. ^ De- 
batable questions  and  "catch”  questions  are  equally 
undesirable The  essential  facts  or  ideas  must  be 
definitely  presented.  On  the  other  hand  the  problem 
or  question  involved  should  not  be  phrased  so  that 
students  can  make  response  on  the  basis  of  a super- 
ficial association.5 

2.  Is  the  wording  simple  and  direct?  The  premise  should 
be  brief^  but  contain  as  much  as  possible  of  the 
essential  statements.5  This  requirement  differs  from 
the  preceding  question  which  refers  to  the  nature  of 
the  ideas  contained  in  the  premise.  This  question 
refers  to  the  manner  in  which  the  ideas  of  the  premise 

^Lang,  Albert  R.  Op.  cit.  p.136 

^Lefever,  D.  Weltey  Op.  cit.  pp.  410-411 

^Anderson,  H.  R.  and  Lindquist,  E.  P.  "The  Improvement  of 
Objective  Testing  in  History",  Second  Yearbook  of  the  National 
Council  for  Social  Studies,  McKinley  Publishing  Company, 
Philadelphia,  1932,  p.  104 

4 

Lang,  Albert  R.  Loc . cit. 
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Oreene,  Harry  A.,  Jorgensen,  Albert  N.,  and 
Oerberich,  J.  Loc.  cit. 
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are  expressed. 

3 . Is  the  premise  expressed  In  the  form  of  an  incomplete 
statement  or  as  a question?  Both  methods  of  presen- 
tation are  suggested  and  discussed  by  various  writers 
but  no  preference  is  shown  for  either  one.  The  direct 
question  may  be  more  easily  phrased.-'-  Because  both 
types  have  been  used  in  the  items  covered  by  this 
studv  the  two  groups  will  be  compared  on  this  point. 

4 . Is  the  - statement  in  the  premise  positive  or  negative? 
The  use  of  negative  statements  should  be  avoided 
according  to  Lang.^  This  study  will  not  throw  much 
light  on  the  point  because  negatives  were  not  commonly 
used  in  the  tests. 

5.  Is  the  premise  so  phrased  that  the  "best  answer" 
rather  than  the  "correct  answer"  is  called  for? 
Although  no  one  has  expressed  an  opinion  on  the  rela- 
tive value  of  the  two  types  of  items  suggested  in  the 
question,  it  is  apparent  that  the  two  approaches  may 
show  differences  in  validity.  Consequently  this  Doint 
will  be  explored. 

6 . Is  the  premise  a statement  of  a single  idea  or  does 
it  Involve  a combination  or  contrast  of  ideas?  There 

l-Hawkes,  Herbert  E.,  Lindquist,  E.  F.,  and  Mann,  C.R.  The 
Construction  and  Use  of  Achievement  Tests,  Houghton  Mifflin 
Company,  Eoston,  1936,  p.  139 

p 

Lang,  Albert  R.  Loc . cit. 
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is  the  possibility  that  a premise,  complex  in  its 
content,  may  cause  confusion  in  the  mind  of  the  stu- 
dent, a confusion  that  has  nothing  to  do  with  the 
deliberate  confusion  created  by  the  five  possible 
responses . 

7 . Is  the  nature  of  the  premise  such  that  other  answers, 
as  good  as  any  included  among  the  five  responses, 
require  the  use  of  "of  the  following”?  It  seems  wise 
to  determine  whether  items  of  this  type  are  of  high 
or  low  validity.  They  are  not  uncommon  and  it  has 
apparently  been  taken  for  granted  that  there  was  no 
difference  in  validity  between  them  and  the  more 
conventional  type  of  item. 

8.  How  many  words  are  used  in  stating  the  premise?  Even 
a casual  inspection  of  almost  any  multiple-choice  test 
reveals  the  fact  that  items  differ  greatly  in  respect 
to  the  length  of  the  premise.  Some  consist  of  only  a 
single  brief  sentence.  Others  are  of  paragraph 
length.  The  simplest  objective  way  to  compare  items 
on  this  point  is  to  count  the  number  of  words  used  in 
the  premise. 

The  correct  answer.  The  nature  of  the  correct  answer  has 
received  its  share  of  attention  from  those  who  write  tests  and 
those  who  write  about  them. 
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1 . Is  there  only  one  correct  answer  to  satisfy  the 
premise,  an  answer  on  which  competent  authorities 
will  agree?  In  preparing  multiple-choice  items,  it  is 
a prime  essential,  emphasized  by  Rinsland,^  that  there 
be  only  one  correct  answer,  that  the  correctness  be 
established  beyond  question,  and  that  the  correct 
answer  be  included  among  the  responses. 

2 . Is  the  answer  so  worded  that  it  can  be  determined 
without  reference  to  the  premise?  It  is  important 
that  the  incorrect  answers  be  not  too  evidently 

incorrect.* 2  If  the  premise  is  in  the  form  of  an 
incomplete  statement  no  single  answer  can  stand 
alone.  If  the  premise  is  in  the  form  of  a direct 
question,  it  is  possible  that  answers  in  the  form 
of  statements  misht  be  so  complete  that  the  correct 
response  would  suggest  itself  to  the  student  without 
requiring  reference  to  the  premise.  As  remote  as 
this  possibility  seems  to  be,  the  items  will  be 
studied  with  this  point  in  mind. 

3 . Is  the  correct  answer  indicated  by  outward  characteris- 
tics such  as  length,  or  repetition  of  words  in  the 
premise?  Any  characteristic  that  tends  to  attract 

^Rinsland,  Henry  D.  Loc . cit. 

20’Dell,  C.  W.'  Traditional  Examinations  and  New  Type  Tests, 
Century  Company,  New  York,  1928,  p.  249 
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attention  to  the  correct  answer  affects  the  validity 
of  the  item.-*-  In  some  cases,  a desire  to  he  accurate 
in  phrasing  causes  the  item  writer  to  make  the  correct 
choice  longer  than  the  distracters.  Length  in  itself 
may  be  a signal  to  those  who  guess.  Careless  phrasing 
may  result  in  repetition  of  parts  of  the  premise, 
another  signal  to  the  unenlightened. 

Is  the  correct  or  incorrect  answer  called  for?  In  a 
test  which  includes  an  appreciable  number  of  items  it 
is  assumed  that  for  the  most  part  items  would  call 
for  the  correct  answer.  The  examinee  becomes  accus- 
tomed to  the  pattern.  If  here  and  there  throughout 
a test  an  item  calling  for  the  incorrect  answer  is  in- 
troduced, the  student  is  confused  not  by  the  item  but 
in  the  change  from  the  positive  to  the  negative  nature 
of  the  problem.  A loss  of  item  validity,  it  is 
claimed,  will  result. 

How  many  words  are  there  in  the  correct  answer?  The 
emphasis  placed  on  brevity  by  Rinsland2  among  others 
indicates  that  subjectively  this  is  considered  a vital 
Doint.  The  wordiness  of  the  correct  answer  will  be 
observed  to  determine  its  effect  on  validity. 

Is  the  correct  answer  simple  or  complex?  A frequently 
observed  difference  in  correct  answers  is  that  which 


^Rinsland,  Henry  D.  Op.  cit.  p.20 
^Rinsland,  Henry  D.  Op.  cit.  p.21 
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exists  between  responses  which  contain  a single 
idea  and  those  consisting  of  a combination  or 
contrast  of  several  ideas.  Because  the  possi- 
bility exists  that  this  simplicity  or  complexi- 
ty in  the  content  of  the  correct  answer  may 
affect  validity,  this  point  will  be  studied. 

The  two  types  of  items  are  illustrated  in 
Figures  6 and  7. 


If  warm  moist  air  is  passing  over  colder 
ground  and  the  wind  velocity  is  25  mph, 
which  of  the  following  clouds  is  likely 
to  form? 

a . Cumulus . 

b . Cumulonimbus . 

c.  Stratocumulus . 

d.  Fog. 

e.  Altostratus  . 

Figure  6 . 

An  Item  Containing  Simple  Responses 
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The 

for 

four  items  needed  for  vector  solution 
TH  and  OS  are: 

a. 

TC, 

TAS, 

track,  wind  direction. 

b. 

TC, 

TAS, 

wind  direction,  wind  speed. 

c . 

TC, 

TAS, 

track,  wind  speed. 

d. 

TC, 

TAS, 

drift  correction,  wind  speed. 

e . 

None  of 

the  above  answers . 

Figure  7. 

Item  Containing  Complex  Responses 


The  distracters . Much  has  been  written  about  the  impor- 
tance of  the  distracters.^  Their  preparation  requires  the 
maximum  care  reflected  in  the  following  questions. 

1 . Is  each  so  plausible  that  one  not  knowing  the  correct 
answer  will  choose  it?  Responses  that  are  plausible 
should  be  used.'5  The  more  plausible  the  distracters 
the  smaller  the  effect  of  chance.3  They  should  not 
be  so  plausible  however  that  they  become  negatively 
discriminating. ^ An  obviously  ridiculous  or  in- 
consistent distracter  presents  no  challenge  to  the 
student . 

^McNamara,  W.  J.  and  Weitzman,  E.  Loc . cit. 

^Lincoln,  E.  A.  and  Workman,  L.  L.  Testing  and  the  Use  of 
Test  Results,  The  Macmillan  Company,  New  York,  1935,  p.  201 

3Horst,  Paul  "The  Difficulty  of  a Multiple-Choice  Test 
Item",  Journal  of  Educational  Psychology,  Vol.  24,  pp.  229-232, 
March,  1923 

^Hawkes,  Herbert  E.,  Lindquist,  E.  F.,  and  Mann,  C.  R. 

Op.  cit.  p.143 
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2 .  Are  each  related  to  the  premise  Ideologically?  The 


necessity  for  avoiding  the  use  of  irrelevant  clues  is 
clear.1 2 3  All  responses  should  be  consistent  with  the 
premise  so  that  none  can  be  excluded  by  a simple 
inspection  of  the  item.  Each  distracter  should,  to 
the  uninformed,  constitute  a logical  answer. 

3.  Are  each  related  to  the  premise  grammatically?  There 
should  not  be  any  observable  grammatical  inconsistency 
between  the  premise  and  each  distracter.  The  con- 
struction should  be  such  that  each  response,  in 
connection  with  the  premise,  constitutes  a complete 
unit  grammatically.^ 

4.  Are  distracters  too  close  to  correct  answer?  In  the 
task  of  selecting  the  right  "wrong  answers"  there  is 
the  danger  of  selecting  misleads  so  close  in  meaning 
to  the  correct  answer  that  only  academic  hair- 
splitting separates  the  false  from  the  true.  If  the 
problem  is  mathematical,  the  misleads  should  be  far 
enough  from  the  correct  answer  to  prevent  the  ordi- 
nary "rounding  off"  process  from  leading  the  student 
to  the  wrong  choice.  Distinction  between  the  right 

1Anderson,  H.  R.  and  Lindquist,  E.  F.  Op.  cit.  p.104 

2Hawkes,  Herbert  E.,  Lindquist,  E.  F.,  and  Mann,  C.  R. 

Op.  cit.  pp.  140-142 

✓ 

30reene,  Harry  A.,  Jorgensen,  Albert  N.,  and  Oerberich,  J. 

Op.  cit.  p.  193 
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and  the  wrong  answer  should  not  depend  on  fine  shades 
of  word  meaning. 

5.  Are  the  dlstracters  of  approximately  the  same  length 
as  the  correct  answer?  As  stated  in  the  discussion 
of  the  correct  answer,  the  nicety  of  phrasing  in  the 
correct  response  often  results  in  the  use  of  a 
greater  number  of  words  than  those  used  to  phrase  the 
misleads.  The  effect  on  validity  of  these  differences 
in  length  will  be  noted. 

6 . Are  dlstracters  parallel  in  form  to  the  correct 
answer?  In  the  preparation  of  misleads  the  aim  is, 
of  course,  to  have  the  misleads  differ  in  nothing 
but  their  intrinsic  meaning  from  the  correct  answer. 
For  this  reason  it  is  suggested  that,  in  so  far  as 
possible,  all  misleads  should  be  stated  in  the  same 
form  as  the  correct  response. 

7 . Are  dlstracters  parallel  in  meaning  to  the  correct 
answer?  As  they  are  parallel  with  the  correct 
response  in  form  so  should  they  be  in  meaning. 
Relative  freedom  from  obvious  clues  is  thus  assured. 
An  illustration  of  this  similarity  in  form  and 
meaning  follows  in  which  "c"  is  the  correct 


answer 
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To  determine  air  position  by  means  of  air  plot 

you  need  a record  of? 

a.  TH,  TAS,  GS,  and  time  on  each  heading. 

b.  TC,  TH,  and  time  of  departure. 

c.  TH,  TAS, and  time  on  each  heading. 

d.  TH,  TAS,  TC,  and  time  on  each  heading. 

e.  TH,  TAS,  and  average  wind  for  all  heading. 

Figure  8. 

Item  Containing  Responses 
Parallel  in  Form  and  Meaning 

8 • Do  the  distracters,  each  within  themselves, 

repeat  conditions  which  might  have  been  included 
in  the  premise?  Needless  repetition  with  each 
response  of  phrases  which  might  have  been 
included  in  the  premise  results  in  wordiness 
which  destroys  the  brevity  so  emphatically 
demanded  by  many  writers  on  the  subject.  The 
effect  of  such  wordiness  on  validity  will  be 
determined.  The  following  item  illustrates  in  the 
repetition  of  the  "g"  load  phrase  an  awkward  and 
wordy  arrangement  of  material  in  the  misleads.  The 
item  shows  other  undesirable  characteristics  as 
well.  It  is  apparent  that  the  entire  item  should 
have  been  presented  in  a more  direct  manner. 
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An  airplane  is  making  a normal  sideward 
turn  at  100  mph  with  a radius  of  1000  ft. 

Another  airnlane  is  coming  out  of  a dive 
at  100  mph  with  a radius  of  1000  ft. 

a.  The  airplane  making  the  normal 
sideward  turn  has  a greater  "g"  load. 

b.  Both  airplanes  have  the  same  "g"  load. 

c.  The  airplane  coming  out  of  a dive  has 
a greater  "g"  load. 

d.  The  "g"  load  is  not  noticeable  in 
either  airplane. 

e.  If  the  turns  are  made  properly 
neither  airplane  produces  a "g"  load. 

■ i 

Figure  9 

Item  Containing  "Wordy"  Responses 

9 . Is  the  statement  "None  of  these  answers"  or  "Correct 
answer  not  given"  included  among  the  dlstracters? 

The  use  of  these  phrases  is  sometimes  necessary  and 
frequently  desirable  because  of  the  nature  of  the 
item.  The  inclusion  of  such  a mislead  requires  the 
student  to  examine  carefully  the  other  four  alter- 
natives before  he  selects  the  blanket  denial  as  his 
choice.  The  nature  of  the  material  may  be  such  that 
four  satisfactory  misleads  cannot  be  readily  devised. 
In  such  cases  the  inclusion  of  "None  of  these 
answers"  makes  it  possible  to  preserve  the  uniformity 

of  the  five  point  structure.  There  is  evidence, 

✓ 

though  somewhat  inconclusive,  that  the  use  of  such 
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misleads  does  not  produce  undesirable  results.  ^ 

10 • Do  the  dlstracters  (as  well  as  the  correct  answer) 
consist  of  single  ideas,  or  do  they  each  present  a 
combination  of  ideas  for  comparison  or  contrast? 

The  two  items  shown  in  Figures  6 and  7 (pp. 22-23) 
illustrate  the  difference  between  the  simple  and 
complex  choices.  It  is  apparent  that  there  should 
be  some  difference  in  the  results  obtained  from  the 
two  types.  In  the  first  (Figure  6)  a single  idea  is 
stated  in  each  mislead.  In  the  second  (Figure  7)  all 
of  the  various  factors  in  the  several  responses  must 
be  considered  in  relation  to  the  premise.  This 
difference  in  the  two  items  suggests  a possible 
difference  in  validity. 

11 . Are  there  "specific  determiners"  included  in  the 
distracters?  The  use  of  "always"  and  "never" 
usually  labels  the  responses  in  which  they  occur  as 
incorrect.  Such  specific  terms  should  not  be  employed 
unless  required  by  the  nature  of  the  material.2 

desman,  A.  G.  and  Bennett,  G.  K.  "Use  of  ’None  of  these* 
as  an  Option  in  Test  Construction",  Journal  of  Educational 
Psychology,  Vol.  37,  pp.  541-549,  December,  1946 

^Ruch,  G.  M.  "Recent  Developments  in  Statistical  Procedures", 
Review  of  Educational  Research,  Vol.  3,  pp . 33-40,  February, 

1933 
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Although  test  makers  long  ago  learned  to  avoid 
such  terminology,  such  words  appear  occasionally 
and  their  effect  on  validity  should  he  determined. 

SUMMARY 

The  foregoing  analysis  of  current  thought  on  the  question 
of  item  preparation  has  resulted  in  a check  list  containing 
thirty-seven  specific  points.  The  remainder  of  this  study 
will  he  concerned  with  an  evaluation  of  these  points  carried 
out  according  to  the  plan  outlined  in  the  following  chapter. 
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CHAPTER  III 

THE  DESIGN  OP  THE  STUDY 
I.  INDEX  OP  VALIDITY 

The  importance  of  item  validity  in  this  study . Thi s s tudy 
in  broad  outline  consists  of  three  phases,  one  of  which,  a 
summary  in  check  list  form  of  current  thought  on  item  writing, 
has  already  been  completed.  The  remaining  phases  will  be 
discussed  later- in  detail.  Because  they  are  almost  exclusively 
concerned  with  statistically  computed  indices  of  validity,  it 
is  necessary  to  explain  what  index  is  to  be  used,  how  it  is 
obtained,  how  its  significance  is  determined. 

The  phi  coefficient.  The  phi  coefficient  will  be  used  as 
the  index  of  validity  throughout  the  study.  The  use  of  this 
index  has  been  justified  by  Guilford  in  his  report  "The  Phi 
Coefficient  and  Chi  Square  as  Indices  of  Item  Validity"^  and 
presented  in  text-book  form  in  Pundamental  Statistics  in 
Psychology  and  Education. ^ It  compares  favorably  with  biserial 
r and  other  similar  measures  of  relationship.  In  the  third 
place  it  is  computed  with  much  greater  ease  than  other  coeffi- 
cients, a most  important  consideration  in  deeling  with  a large 
number  of  items. 

■^Guilford,  J.  P.  Loc . cit. 

^ -Pundamental  Statistics  in  Psychology  and 

Education,  McGraw-Hill  Book  Company,  New  York,  1942,  pp. 296-298 


Ill  3f;  ) 

YdUTc  Ir  HT  YO  f.CU  »C  FI  T 

YTI  I_UV  ' 0 X C - I .1 

. 

t tn6«r!iJ0 

o neatf  vJbseTls  aad ; 

V©f<t  6?.JJ80- 

. 

‘ ‘ t 

. ‘ I 

' 

. 

L ' . bn  i '^oIorioYB^1. 

. 

eoo  ri  • to  t ? - on  1 inr^o'y  ' r I ©o  ;Iq 

* • • d <'i  nolivi?"  nco  J trroq'  t < , : lo 

. '•  • ' ■i'lfVjrn 

. tt  o .OcJ  . i . Tt  , run'll 

. 


31 


Method  of  determining  phi.  Phi  is  determined  by  substi- 
tuting in  the  following  formula: 

t - Pu  - Pi 
2V 

Pu,  the  percentage  of  the  upper  group  passing  the  item.  Pi,  the 
percentage  of  the  lower  group  passing,  p,  the  percentage  of  the 
total  group  passing,  and  q equal  to  1-p.  The  upper  and  lower 
groups  consist  respectively  of  the  highest  and  lowest  25  per 
cent  of  total  scores  on  a given  test. 

Phi  may  be  .more  easily  obtained  by  referring  to  an  abac 
prepared  by  Guilford  in  which  the  value  of  phi  is  determined  by 
the  intersection  of  the  Pu  and  Pi  axes.  Care  must  be  used  in 
reading  phi  values  based  on  extremely  high  or  low  per  cents. 

Significant  scores.  According  to  Guilford,  phi  is  signifi- 
cant at  the  five  per  cent  level  (using  the  tables  for  signifi- 
cance of  chi-square)  and  the  following  formula  is  recommended  to 
determine  a significant  $ score  for  any  given  number  of  cases: 

Significant  jzf  = y 3,941 

N 

In  the  case  of  most  of  these  items,  the  phi  scores  were  computed 
on  a basis  of  400  responses.  A phi  score  of  .10  is  therefore 
significant.  As  N decreases,  the  size  of  phi  to  be  significant 
must  necessarily  increase.  For  example,  when  N - 200,  phi  must 
be  .13  to  be  significant.  When  N = 100,  0 must  be  .20  or 
greater . 

A very  significant  phi  at  the  one  per  cent  level  is 
determined  by  the'  formula, 

^ ' Y 6.535 
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When  N = 400,  0 must  be  .13.  When  N = 200,  0 = .17,  and 
when  N = 100,  0 = .26. 

Index  of  ease.  It  may  appear  unnecessary  to  point  out 
that  items  differ  in  respect  to  difficulty.  A discussion  of 
the  relative  merits  of  easy  and  difficult  items  is  not  perti- 
nent at  this  point.  It  is  necessary,  however,  to  recognize 
the  fact  that  the  difference  exists  and  that  this  difference 
may  be  important  in  later  phases  of  this  study.  Consequently, 
the  Index  of  Ease  for  each  item  was  computed  by  finding  the 
average  of  the  percent  of  the  upper  group  and  the  per  cent  of 
the  lower  group  passing  the  item. 

II.  EVALUATION  OP  PRESENT  DAY  GUIDES 

The  test  of  experience.  The  value  of  the  points  con- 
sidered essential  in  item  writing  which  were  outlined  in  the 
preceding  chapter  will  be  subjected  to  two  tests.  The  first 
of  these  tests  will  consist  of  measuring  the  degree  of 
difference  on  each  point  between  items  of  known  high  validity 
and  those  of  equally  well  established  low  validity.  Points 
will  be  retained  in  or  rejected  from  the  check  list  on  the 
basis  of  their  apparent  contribution  to  item  validity. 

Source  and  nature  of  items  of  established  validity.  The 

large  number  of  unrevised  items  prepared  for  use  in  the 

achievement  testing  program  in  the  AAP  Pilot  Training  Ground 

Schools  provides  a wealth  of  material  with  which  to  test  the 

✓ 

value  of  the  points  in  the  check  list  as  representing  possible 
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distinguishing  characteristics  between  high  and  low  validity 
items.  There  were  four  examinations  consisting  of  the  indicat- 
ed number  of  items  for  each  of  the  sixteen  courses  listed  below: 


Name  of  Course  No.  of  Items  Total  Items 

per  test 


Maps,  Charts,  and  Aerial  Photos 

20 

80 

Applied  Aero  Mathematics 

40 

160 

Aero  Equipment 

Primary 

40 

160 

Single  Engine  Advanced 

40 

160 

Two  Engine  Advanced 

40 

160 

Aircraft  and  Principles  of  Plight 

50 

200 

Navigation  - 

Primary 

30 

120 

Basic 

30 

120 

Advanced 

30 

120 

Instruments  of  Plight 

Basic 

25 

100 

Advanced 

15 

60 

Weather 

Basic 

50 

200 

Advanced 

50 

200 

Radio  Communications 

20 

80 

Armament 

20 

80 

Gunnery 

20 

80 

Total  Items 

The  sixtv-four  examinations  included  in  this 

2080 

studv  pro- 

vided  a net  total  of  2075  items  for  which  the  Index  of  Validity 
was  computed.  As  shown  in  Table  1 (Appendix  B,  page84)  these 
indices  ranged  from  +.59  to  -.29.  The  mean  phi  for  all  items 
was  22.5,  S.D.  .13. 

Selection  of  criterion  groups.  Whatever  distinction  be- 
tween items  of  high  validity  and  low  validity  may  be  discovered 
as  this  study  continues  will  be  found  between  a group  of  items 
whose  mean  Index  of  Validity  is  .48  (range  .70  - .41)  as  shown 
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in  Table  2 (Appendix  B,  page  85)  and  a group  of  those  whose 
mean  phi  score  is  .045  (range  +.10  to  -.22)  from  which  certain 
exclusions  have  been  made. 

An  examination  of  the  items  comprising  these  groups  shows 
that  certain  items  must  be  excluded  if  the  comparison  is  to  be 
meaningful.  In  the  higher  group,  there  are  many  items  whose 
index  of  validity  is  obviously  high  because  of  the  dispropor- 
tionately small  number  of  responses  made  by  the  lower  group  of 
students.  Because  phi  reflects  the  difference  between  the 
percentage  of  responses  of  the  two  groups  of  students,  the  index 
is  high  when  a relatively  small  percentage  of  the  lower  group 
makes  the  correct  resoonse.  In  the  case  of  items  which,  be- 
cause of  the  length  or  difficulty  of  the  test  as  a whole,  very 
few  of  the  poorer  students  attempted  to  answer,  phi  based  on 
the  total  group  will  be  misleadingly  high.  A second  phi  was 
calculated  based  on  the  number  answering  the  question  without 
reference  to  the  size  of  the  total  group.  All  items  whose  phi 
scores  based  on  the  number  of  cases  answering  the  item  were 
below  the  level  of  significance  were  excluded,  although  their 
phi  scores  based  on  the  total  group  were  very  significantly 
high . 

In  studying  the  items  that  fell  more  than  one  sigma  below 
the  mean,  it  was  observed  that  many  which  had  a low  Index  of 
Validity  had  a very  high  Index  of  Ease.  This  phenomenon  can  be 
expected  in  achievement  tests  designed  not  only  to  distinguish 
the  achievers  from  the  non-achievers  but  to  provide  for  adequate 
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course  coverage  as  well.  There  was  a definite  effort  made  in 
the  construction  of  these  tests  to  cover  all  essential  parts  of 
instruction  in  the  course.  As  a result  there  were  many  items 
which  all  or  nearly  all  were  able  to  answer  correctly.  The 
Indices  of  Validity  for  such  items  are  low  for  reasons  other 
than  faulty  construction.  For  this  reason  in  selecting  items 
for  this  study  it  was  necessary  to  discard  those  whose  low 
validity  was  obviously  due  to  ease.  Consequently  every  item 
with  an  Index  of  Validity  of  .10  or  less  with  an  Index  of  Ease 
greater  than  .85  was  eliminated  from  the  study.  There  remained 
after  this  exclusion  had  been  made  one  hundred  sixty  items  of 
the  original  three  hundred  fifty  seven  whose  Index  of  Validity 
was  below  the  level  of  significance.  The  phi  distribution  for 
these  items  is  shown  in  Table  2 (Appendix  B,  page  85  ) . 

Because  one  hundred  sixty  items  now  comprised  the  low 
validity  group,  an  equal  number  of  high  validity  items  were 
selected,  after  the  exclusion  previously  described  had  been  made. 
The  distribution  of  phi  scores  for  this  criterion  group  is 
shown  in  Table  3 (Apoendix  B,  page  85  ) . 

The  need  for  objectivity  in  evaluation.  An  examination  of 

the  check  list  will  show  several  points  on  which  the  evaluation 

naturally  is  completely  objective.  The  answers  to  such 

questions  as  those  concerning  the  negative  or  positive  nature 

of  the  premise,  grammatical  correctness,  number  of  words,  and 
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the  use  of  specific  ohrases  are  obvious.  Of  the  thirty-seven 
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questions  in  the  list,  thirty  are  of  this  type. 

The  answers  to  the  seven  remaining  questions  cannot  be 
answered  by  a simple  examination  of  the  items.  Such  questions 
as  "Does  the  premise,  combined  with  the  right  answer,  consti- 
tute a definite  and  true  concept?"  and  "Is  a definite  task 
clearly  and  unambiguously  set?"  depend  upon  individual  judge- 
ment and  opinion  for  their  answers.  In  order  to  attain  as  high 
a degree  of  objectivity  as  possible  in  evaluating  such  points, 
the  opinion  of  every  instructor  who  had  used  the  tests  was 
considered.  When  the  original  tests  were  given  from  which  the 
items  in  this  study  were  drawn,  e£ch  instructor  was  requested 
to  criticize  each  item  on  a form  provided  for  the  purpose. 

These  criticisms  were  available  (15-25  for  eech  test)  and  used 
in  arriving  at  the  answer  called  for  in  the  check  list.  If 
more  than  20 % of  the  judgements  were  negative  on  a given  point, 
the  item  was  considered  unsatisfactory  in  this  respect. 

The  significance  of  observed  differences.  The  results  of 

this  analysis  will  be  studied  statistically  to  determine  on 

what  points,  if  any,  there  may  be  a significant  difference 

between  the  two  criterion  groups.  The  chi-square  technique1 

will  be  used  to  determine  the  degree  to  which  these  differences 

may  be  significant.  The  hypothesis  is  made  that  there  is  no 

difference  between  the  two  groups  and  then  the  degree  of 

likelihood  that  the  samples  could  have  diverged  from  this 

✓ 

1Guilford,  J.  P.,  Op.  cit.  pp.  167-173 
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hypothesis  as  much  as  it  did  had  chance  factors  alone  been 

operating,  is  determined.  It  is  a measure  of  the  amount  of 

discrepancy  between  hypothesis  and  observation. 

One  advantage  of  chi-square  is  that  it  can  be  computed 

directly  from  proportions  and  its  significance  readily 

interpreted  by  means  of  tables. 

Chi-square  = N ( a-b)^  when  calculated  from  proportions 

pq 

with  a and  b the  proportions  indicated  in  the  cells  below. 


Upper 

Lower 

Group 

Group 

Characteristics 

Present 

a 

b 

Characteristics 

Absent 

c 

d 

.50 

. 

Ol 

o 

A chi-square  of  3. 841  shows  that  the  observed 
differences  could  occur  because  of  chance  only  five  in  100 
times,  and  that  this  difference  is  therefore  significant. 

A chi-square  of  6.635  shows  that  the  difference  could  occur 
only  once  in  100  times  and  is  therefore  very  significant. 

In  the  evaluation  of  the  check  list,  those  points  will  be 
rejected  for  which  the  difference  between  high  and  low  validity 
is  found  to  be  below  the  level  of  significance,  i.e.  the  5% 
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III.  EXPERIMENTAL  REVISION  OP  ITEMS 
OP  UNDETERMINED  VALIDITY 

The  third  phase  of  the  study.  The  points  in  the  check 
list  on  which  items  of  high  and  low  validity  are  found  to 
differ  significantly  will  be  used  as  guides  in  revising  items 
in  tests  designed  for  two  distinct  educational  levels.  The 
items  used  in  the  previously  outlined  evaluation  were  designed 
for  AAP  pilot  trainees  who  in  educational  achievement  and  age 
are  for  the  most  part  comparable  to  college  freshman.  The 
first  test  to  be  revised  will  be  one  for  use  with  first-year 
college  students.  In  order  to  test  the  check  list  for  a more 
universal  application,  a similar  experiment  will  be  conducted 
at  the  junior  high  school  level.  In  each  case  tests  designed 
for  practical  use  will  be  administered. 

A test  for  college  freshman  (Test  No.  1) . An  instructor- 
made  test  consisting  of  fifty-five  untried  multiple-choice 
items  covering  the  first  month's  work  of  the  second  semester 
in  a course  in  Human  Relations  will  be  examined  with  the 
revised  check  list  as  a guide.  Revisions  will  be  made  wherever 
a need  is  indicated.  The  large  number  of  students,  more  than 
seven  hundred,  who  will  take  the  test  will  make  it  possible  to 
get  comparable  results  on  both  forms  of  an  item  without  resort- 
ing to  the  test-retest  technique.  The  original  and  the  revised 
items  will  be  included  in  the  four  different  forms  of  the  test 
in  such  a way  that  statistically  satisfactory  returns  will  be 


. <-;P?  ■'rtf  <?Q  ■ 

i i ->  Hi  ; v cl  jr  it  Irf  lo  • 3.  ’■  rioJ  no  HI 

r : ■ '"■•  :.r  mi  vr^H'  o*  r' ? rr  ■ ’ > 

. 

bn  t ■ ^3  vs v : s iano ' i i orw  *?€•  x.f  JcQg  '*  AA  "c‘: 

. - f r ’ 0 r ' ■ . . . ■ 

c ;c  »d  r ' £ r . - c cd  dtf  1 J?  'll 

. 

■ ■ 

. 

. •■  H ' ' ■ t • -*  r.!  ' ■ r • • o »r  *TC ‘ 


. 

- 

-;*T<  ’ rrc  -T.r  ^rid  T>f  l^  vn  f =*d  ‘ 

. 

’ ’ 

-Hoj  t l ‘ f ' . 'ic  1 c Her  t £ >"  ..  To;  ■ * 

. 

rrr,  ~tP'  ? 1 yfj  ,oj  X d .-t  i.i  ' <?r  1 r)t»i  r 1 


available  for  each  form  of  each  item.  At  least  one  fourth  of 
the  group  will  use  the  original  form  of  each  item  8nd  another 
fourth  will  use  the  revised  form.  Each  student  will  respond 
to  an  approximately  equal  number  of  original  and  revised 
items.  This  test  will  hereafter  be  referred  to  as  Test  No.  1. 

A test  for  junior  high  school  students  (Test  No.  2).  The 
second  test  to  be  revised  experimentally  will  consist  of  items 
similar  to  those  in  the  multiple-choice  section  of  the  test 
in  History  and  Civics  included  in  an  early  form  of  the 
Metropolitan  Achievement  Test.-*-  These  items  will  be  expanded 
from  the  four-choice  to  the  five-choice  form  by  the  addition 
of  a single  appropriate  distracter.  The  items  will  then  be 
revised  using  the  revised  check  list  as  a guide.  Two  forms  of 
the  test  will  be  set  up.  One  will  include  the  original  form 
of  the  odd  numbered  items  and  the  revised  form  of  the  even 
numbered  items.  The  other  will  include  the  even  numbered 
original  items  and  the  odd  numbered  revised  items.  Each  of  the 
two  forms  will  be  given  to  an  equal  number  of  students  for  the 
first  administration  of  the  test.  Two  days  later  the  test  will 
be  repeated.  Those  pupils  who  used  the  first  form  on  the  first 
day  will  use  the  second  form  and  those  who  at  first  used  the 
second  will  complete  the  first  form.  A response  to  each  form 

1 Allen,  R.  D.,  Bixler,  H.  H.,  Connor,  W.  L.,  and  Graham,  F.B., 
Metropolitan  Achievement  Tests,  Advanced  Battery-Complete: 

Form  A for  Grades  H and  9 (Revised),  World  Book  Company. 
Yonkers-on-Hudson,  New  York,  1933,  pp.  22-25 
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of  each  item  will  be  obtained  from  each  student  in  such  a way 
that  any  effect  of  practice  or  fatigue  on  the  results  will  be 
counteracted.  The  number  of  pupils  to  be  tested  is  approxi- 
mately one  hundred  fifty.  This  test  will  hereafter  be  referred 
to  as  Test  No.  2. 

Interpreting  the  results  of  revision.  A phi  coefficient 
of  validity  will  be  obtained  for  both  forms  of  each  item.  To 
whatever  differences  invalidity  may  be  observed  between  the  two 
forms,  the  test  of  significance  for  Fisher’s  null  hypothesis 
will  be  applied.  The  formula  to  be  used  is  adapted  from 
Peatman.-*- 

t = Mr  - M o 

V"  S.D.r£  + S.D.0z 

The  revised  suggestion!  for  item  writing  as  9 whole  will 
be  accented  as  having  effected  a satisfactory  increase  in 
validity  if  the  degree  of  difference  between  the  mean  phi 
of  the  revised  items  and  that  of  the  original  items  reaches  or 
exceeds  the  5 % level  of  significance. 

The  mean  phi  scores  of  items  revised  in  accordance  with 
the  provisions  of  a single  point  in  the  check  list  and  the 
mean  phi  scores  of  the  original  form  of  these  items  must  show 
a similar  difference  if  that  particular  point  is  to  be  con- 
sidered effective. 

^Peatman,  J.  G.  Descriptive  end  Sampling  Statistics.  Harper 
and  Brothers,  New  York,  1947,  pp.  397-399  and  409-410 
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The  chi-square  technique  for  determining-  the  significance 
of  differences  will  not  be  employed  in  this  phase  of  the  study 
because  it  is  assumed  that,  due  to  the  number  of  items  in  each 
test,  frequencies  in  the  various  distributions  of  phi  will  be 
smaller  than  those  indicated  by  Peatman^  as  essential  for  its 
effective  use. 

It  is  quite  probable  that  in  the  course  of  the  experiment 
items  will  be  found  which  will  not  need  revision.  These  items 
will  be  retained -in  the  tests  and  the  resulting  phi  scores 
studied  for  whatever  significant  information  may  be  revealed. 

If  there  are  enough  of  them  in  one  test  to  sive  statistically 
reliable  results,  a comparison  may  be  possible  of  the  mean 
phi  * s of  the  unrevised,  the  original  and  the  revised  items. 

The  difference  between  the  mean  phi  scores  of  the  items  selected 
for  revision  and  the  mean  phi  of  those  requiring  no  revision 
may  be  an  indication  of  the  value  of  the  check  list  as  a 
screening  device  in  the  selection  of  test  items. 

SUMMARY 

The  three  phases  of  this  study  consist  of  (1)  a summary  of 
current  thought  in  the  form  of  a check  list  of  points  to  be 
considered  in  item  writing;  (2)  a study  of  items  of  high  end 
low  validity  selected  from  a total  of  more  than  two  thousand 
items  of  established  validity  in  order  to  delete  from  the  check 
list  those  points  on  which  the  hiffh  and  low  items  do  not  differ 

^-Op.  cit.  p.424 
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signif icantly;  (3)  an  experiment  to  determine  how  effective 
the  remaining  points  may  be  in  increasing  item  validity. 
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CHAPTER  IV 

THE  EVALUATION  OF  PRESENT  DAY  GUIDES 
I.  THE  EVALUATION  TECHNIQUE 

The  check  list  in  use.  The  general  method  for  conducting 
this  phase  of  the  study  has  been  outlined  in  the  preceding 
chapter.  The  selection  of  the  criterion  groups  of  items  has 
been  described  in  detail.  Each  of  the  one  hundred  sixty  items 
in  each  of  the  two  criterion  groups  will  now  be  examined  on 
every  point  in  the  check  list  and  the  results  tabulated.  (See 
Table  4,  Appendix  B,  pages  86-88)  The  responses  to  the  questions 
will  apoear  first  as  the  number  of  high  validity  and  the 
number  of  low  validity  items  written  according  to  the  pattern 
described  or  implied  in  each  question. 

Maintaining  objectivity.  It  has  already  been  pointed  out 
that  as  complete  objectivity  as  possible  must  be  maintained 
throughout  the  evaluation.  Of  the  thirty- seven  points  in  the 
check  list,  twenty-six  are  concerned  with  characteristics  the 
presence  or  absence  of  which  is  readily  determined  by  observa- 
tion. No  explanatory  comment  on  these  points  will  be  necessary 
when  the  results  of  the  evaluation  are  presented  in  Parts  II, 
III,  IV,  and  V of  this  chapter.  There  are  eleven  questions  in 
the  list  which  depend  on  more  than  observation  for  answers. 

The  manner  in  which  the  evaluation  was  handled  In  connection 
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with  these  points  was  as  objective  as  the  nature  of  the 
question  permitted  and  will  be  described  in  connection  with 
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each  of  the  points  concerned. 

Determining  the  significance  of  the  differences.  At  each 
point  of  the  check  list  a comparison  of  the  numerical  count  of 
items  of  high  validity  with  those  of  low  validity  shows  that 
the  difference  between  the  two  varies  from  point  to  point.  In 
order  to  determine  the  significance  of  these  varying  differences, 
using  the  chi-square  technique  already  explained  in  detail,  the 
numerical  counts  are  converted  to  percentages  and  these  values 
substituted  in  the  chi-square  formula.  The  level  of  signifi- 
cance of  the  chi-square  value  thus  obtained  is  then  determined. 
It  has  already  been  specified  that  a check  list  point,  to  be 
considered  effective,  must  show  a difference  which  results  in 
a chi-square  whose  level  of  significance  is  no  less  than  5$. 

The  results  of  the  evaluation  follow.  All  data  involved  are 
shown  in  Table  4,  (Anoendix  B,pp.86-B8). 

II.  THE  ITEM  AS  A WHOLE 

1 . Does  the  premise,  combined  with  each  choice,  con- 
stitute a complete  unit  of  thought  ideologically? 


High 

Low 

Chi- 

Level  of 

Validity 

Validity 

Square 

Significance 

Yes  100$ 

luO$ 

0 

99$ 

No  0 

U 

2 . Does  the  premise,  combined  with  each  choice,  constitute 
a complete  unit  of  thought  grammatically? 


' 


- ■ • 


r . 


. 


High  Low  Chi- 

Validity  Validity  Square 


Level  of 
Significance 


Yes  100#  96.2#  5.776 

No  0 3.8 


5# 


Does  the  premise,  combined  with  the  right  answer, 
constitute  a definite  and  true  concept?  The 
criticisms  of  the  instructors  using  the  test  were 
considered  in  answering  this  question.  If  more  than 
20#  of  the  instructors  reporting  considered  an  item 
unsatisfactory  in  this  respect,  the  answer  was  "No" 
on  this  point. 


High 

Validity 


Low  Chi- 

Validity  Square 


Level  of 
Significance 


Yes  100#  93.7#  10.251 

No  0 6.3 


1# 


. Is  the  idea  stated  clearly  and  directly  with  the 
answer  an  important  part  of  the  statement?  The  same 
method  of  using  instructor  comment  as  previously 
described  was  followed. 


High 

Validity 


Low 

Validity 


Chi-  Level  of 

Square  Significance 


Yes  96.9#  85.6#  12.544  .1# 

No  3.1  14.4 


Does  the  item  call  for  problem  solving,  for  interprets- 


tlon  of  data,  or  for  the  recall  of  factual  information? 


. 

f- . 


• 1 

cM 


err.,  * )r>r\J 


. 


ET  l«t  C f ' ' + 

, 

. 


. 


. 

. < r *"0E9fc 


. s 


. 

. 


. 


, ? 

i 1*2:1  IP _ 


.e 


High 

Low 

Chi- 

Level  of 

Validity 

Validity 

Square 

Significance 

Problem 

19 . 5$ 

13.8$ 

9.3 

1$ 

Data 

54.5 

36.9 

Recall 

26.0 

49.4 

In  recognition  of  the  fact  that  purely  mathe- 
matical Items  are  rarely  written  in  multiple-choice 
form  in  order  to  make  the  findings  of  this  study 
useful  in  as  general  a way  as  possible,  it  will  be 
helpful  to  observe  what  significance  can  be  found  in 
the  difference  between  the  two  remaining  groups, 
without  reference  to  those  of  a mathematical  nature. 

High  Low  Chi-  Level  of 

Validity  Validity  Square  Significance 
(129  items) (133  items) 

Data  67.4$  42.8$  16.48  1$ 

Recall  32.6  57.2 

Is  the  item  a direct  measure  of  the  points  of  mental 

attainment  under  consideration?  Objectivity  can  be 

maintained  in  answering  this  question  by  observing 

the  type  of  answer  required.  If  the  use  of  specific 

information  or  skill  is  Involved  in  the  choice  of  the 

correct  answer  it  is  here  considered  a measure  of 

mental  attainment  but  if  the  item  asks  a question 

about  a given  point  the  answer  "No”  was  given  in  the 
✓ 

evaluation.  The  two  types  are  illustrated  by  the 
items  shown  in  Figures  10  and  11.  The  first  item 
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(Figure  10)  calls  for  the  use  of  knowledge  about  the 
altimeter.  The  second  item  (Figure  11)  calls  only 
for  an  understanding  of  the  instrument. 


Failure  to  reset  the  altimeter  from  the  NO 
value  before  the  approach  at  GR  would 
cause  the  altimeter  to  read  about: 

a. 

90 

ft 

. too 

high. 

b. 

900 

ft 

. too 

low. 

c . 

90 

ft 

. too 

low . 

d. 

900 

ft 

. too 

high. 

e . 

60 

ft 

. too 

low . 

Figure  10. 

A "Direct  Measure  of  Attainment"  Item 


The  aneroid  of  the  altimeter  expands 
contracts  because  of: 

and 

a . 

The  weight  of  air  below  the  altimeter. 

b. 

The  weight  of  the  air  above  the 
altimeter. 

c . 

The  density  of  the  air. 

d. 

The  rate  of  'leak1  of  air  into  or 
of  the  aneroid. 

out 

e . 

The  difference  between  static  and 
impact  pressure. 

Figure  11. 

An  Item  not  a "Direct  Measure  of  Attainment" 


High  Low  Chi-  Level  of 

Validity  Validity  Square  Significance 
✓ 

Yes  78.0^  52 • 5%  22.44  . 1 % 

No  22.0  47.5 
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What  is  the 

position 

of  the  i 

correct 

choice? 

High 

Low 

Chi- 

Level  of 

Position 

Validity 

Validity 

Square 

Significance 

1 

19.4# 

13.8# 

1.282 

90# 

2 

23.8 

23.8 

3 

22.5 

23.1 

4 

21.3 

25.6 

5 

13.1 

13.8 

What  is  the 

Index  of 

Ease? 

Index 

High 

Low 

Chi- 

Level  of 

of  Ease 

Validity 

Validity 

Square 

Significance 

81-100 

6.9# 

15.0# 

26.263 

.1# 

61-80 

42.5 

33.8 

41-60 

46.9 

21.9 

21-40 

3.8 

20.0 

1-20 

0.0 

9.4 

Is  the  Item 

free  from  grammatical,  : 

spelling,  and 

typographical  errors 

High 

Low 

Chi- 

Level  of 

Validity 

Validity 

Square 

Significance 

No  errors 

96.8# 

94.4# 

1.152 

30# 

Errors 

3.2 

5.6 

Are  answers 

arranged  horizon 

tally  or  vertically? 

the  items  in  these  tests,  answers  were  arranged 
horizontally  whenever  their  length  would  permit  such 
arrangement.  The  observed  effect  of  this  lay-out  on 
validity  is  quite  definite.  The  question  arises, 

s 

however,  as  to  whether  this  effect  is  due  to  the 
arrangement  of  answers  or  to  the  nature  of  the  subject- 
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11. 


12. 


matter  which  makes  possible  the  use  of  such  brief, 
concise  answers. 


High 

Low 

Chi- 

Level  of 

Validity 

Validity 

Square 

Significance 

Horizontal 

16.3$ 

4.4$ 

12.8 

.1$ 

Vertical 

83.7 

95.6 

Is  the  item 

self-contained  or 

does  it 

require  the  use 

of  reference 

: material 

not  included  in 

the  test? 

- 

High 

Low 

Chi- 

Level  of 

Validity 

Validity 

Square 

Significance 

Self- 

contained 

35.0$ 

57.5$ 

16.340 

• ±/0 

Added 

material 

65.0 

42.5 

How  many  words  are  used  in  stating  .the  item  as  a 


whole? 

Number 

High 

Low 

Chi- 

Level  of 

of  words 

Validity 

Validity 

Square 

Significance 

56+ 

48.1$ 

17.5$ 

25.08 

.1$ 

46-55 

3.2 

14.4 

36-45 

7.5 

30.0 

26-35 

15.0 

19  .4 

- 25 

26.3 

18.8 

In  an 

attempt  t 

o discover  more 

exactly  the  effect 

of  wordiness  on  the 

validity 

of  the 

items  in  the 

criterion  croups  the  fibres  above  were  revised  by 
✓ 

eliminating  those  items  from  the  tabulation  which 
were  not  self-contained  and  whose  total  number  of 
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words  was  56  or  more.  There  were  72  such  items  in 
the  High  Validity  group  and  25  in  the  Low  Validity 
group . 

III.  THE  PREMISE 

1 . Is  a definite  task  clearly  and  unambiguously  set? 
Instructor  comment  was  taken  as  the  suide  to  the 
answer  to  this  question. 


High 

Low 

Chi- 

Level  of 

Validity 

Validity 

Square 

Significance 

Yes  95.6$ 

80.0$ 

17.689 

.1$ 

No  4.4 

20.0 

Is  the  wording  simple  and  direct? 

The  wording  of 

items  was  considered  satisfactory 

in  these  respects 

if  it  was  acceptable  to 

the  instructors  using  the 

tests  . 

High 

Low 

Chi- 

Level  of 

Validity 

Validity 

Square 

Signif ic  ance 

Yes  98.1$ 

91.2$ 

7.840 

1$ 

No  1.9 

8.8 

Is  the  premise 

s expressed 

in  the  form  of  an  Incomplete 

statement  or  as  a question? 


High  Low  Chi-  Level  of 

Validity  Validity  Square  Significance 

Statement  86.9$  78.1$  4.425  5$ 

Question  13.1  21.9 
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5. 


6. 


Is  the  statement  in  the  premise  positive  or  negative? 


High 

Low 

Chi- 

Level  of 

Validity 

Validity 

Square 

Significance 

Positive 

100# 

98.7# 

1.563 

30# 

Negative 

0 

1.3 

The 

evidence  < 

on  this  point  is 

inconclusive.  No 

negatives 

whatever 

were  to 

have  been  used  in  the 

items  in 

these  tests. 

Is  the  premise  so 

phrased  that  the 

"best  answer" 

rather  than  the  "correct  answer"  is 

called  for? 

High 

Low 

Chi- 

Level  of 

Validity 

Validity 

Square 

Significance 

Best 

5.0# 

11.9# 

5.284 

5# 

Correct 

95.0 

88.1 

Is  the  premise  a statement 

of  a single  idea  or  does 

It  Involve  a combination  or  contrast  of  ideas ? 

If  the  premise  asks  a simple  question  involving 
only  a single  idea  it  is  considered  simnle  in  this 
evaluation.  If  It  presents  more  than  one  condition 
or  contrasts  or  compares  ideas  it  is  considered  complex. 


High  Low  Chi-  Level  of 

Validity  Validity  Square  Significance 


Simple  1 21.9# 

Complex  2 14.4 

(No.  of  3r  8.1 

ideas ) 4( +) 55  *6 


30.0#  2.486 

18.1 
8.1 
43.8 


50# 


Boston  UnlvfifTtty 

School  of  Education 
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7 . Is  the  nature  of  the  premise  such  that  other  answers, 
as  good,  as  any  included  among;  the  five  responses, 
require  the  use  of  Hof  the  following"? 


High 

Validity 


Low 

Validity 


Chi- 

Square 


Level  of 
Significance 


Yes  5 .0$  6.9$  .533 

No  95 .0%  93.1 


50$ 


8 . How  many  words  are  used  in  stating  the  premise? 


Number 

High 

Low 

Chi- 

Level  of 

of  words 

Validity 

Validity 

Square 

Significance 

21+ 

53.8 $ 

32 . 5% 

10.677 

5$ 

17-20 

5.0 

15.0 

13-16 

12.5 

21.9 

7-12 

13.1 

16.3 

IV.  THE  CORRECT  ANSWER 

1 . Is  there  only  one  correct  answer  to  satisfy  the 
premise,  an  answer  on  which  competent  authorities 
will  agree?  The  answer  indicated  as  correct  on  the 
scoring  key  was  accepted  as  the  only  correct  answer 
unless  instructors  using  the  tests  found  these 
answers  wrong. 


High 

Low 

Chi- 

Level  of 

Validity 

Validity 

Square 

Significance 

99.4$ 

91.2$ 

10.758 

1$ 

.6 

8.8 

z . 

. 

. 

r. 

. 

. 


, 
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Is  the  answer  so  worded  that  it  can  be  determined 


without  reference  to  the 

premise? 

High 

Low 

Chi- 

Level  of 

Validity 

Validity 

Square  ! 

Significance 

Yes  1.3# 

1.9# 

.512 

50# 

No  98.7 

98.1 

Is  the  correct  answer  indicated  by 

outward  character- 

istics  such 

as  length,  or  repetition  of  words  in  the 

premise? 

High 

Low 

Chi- 

Level  of 

Validity 

Validity 

Square 

Significance 

Yes  3.2# 

4.4# 

.128 

30# 

No  96.8 

95.6 

Is  the  correct  or  incorrect  answer 

called  for? 

High  Low  Chi- 

Level  of 

Validity  Validity  Square  Significance 

Correct 

99.4#  98 

.7#  .512 

50# 

Incorrect 

.1  1 

.3 

How  many  words  are  there 

in  the  correct  answer? 

Number  High  Low  Chi- 
of  words  Validity  Validity  Square 


Level  of 
Significance 


11+ 

9-10 

7-8 

5-6 

-4 


7.5# 

6.5 

6.3 

13.8 

66.3 


9.4# 

5.6 

9.4 

13.8 

61.9 


.312 
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9. 

6 . Is  the  correct  answer  simple  of  complex? 


High  Low  Chi-  Level  of 
Validity  Validity  Square  Significance 

Simple  1 
Complex  2 
(No.  of  3 
ideas)  4(+) 


72 . 5#  72 . 5# 

25. 0 25.0 

1.3  1.9 

1.3  0.6 


.267  99 # 


V.  THE  DISTRACTERS 

1 . Is  each  so  plausible  that  one  not  knowing  the  correct 
answer  will  choose  it?  The  distracters  are  not  con- 
sidered plausible  in  this  evaluation  if  any  of  them 
failed  to  appeal  to  some  student  or  students  as  the 
correct  answer. 


High 

Validity 


Low 

Validity 


Chi- 

Square 


Level  of 
Significance 


Yes  82.5#  83.8#  .096 

No  17.5  16.2 


95# 


2 . Are  each  related  to  the  premise  ideologically? 

Instructor  comment  was  relied  on  for  indications  of 
inconsistency  in  this  respect.  The  distracters  were 
also  examined  for  possible  use  of  deliberately 
misleading  "trick"  or  "catch"  phrases. 


High  Low  Chi- 

Validity  Validity  Square 

100# 

O 


Level  of 
Significance 


Yes  100# 
No  a. 


0 


99# 


. 

. r : . . 1 ■ 


. 

. 

. 

4 . - 

. 

. 

3 . Are  each  related,  to  the  premise  grammatically? 


High 

Validity 

Yes  98 .7$ 
No  1.3 


Low 

Validity 

99.4# 

.6 


Chi- 

Square 

.512 


Level  of 
Significance 

50# 


4 • Are  distracters  too  close  to  correct  answer? 

Instructor  comment  revealed  instances  of  this  sort. 

The  item  analysis  also  showed  when  a single  distracter 
proved  -appealing  to  a large  number  of  students. 


High 

Validity 


Yes  3.b7o 
No  96.2 


Low 

Validi ty 

16  .9# 
83.1 


Chi-  Level  of 

Square  Significance 


15.022 


.1# 


5 . Are  distracters  of  approximately  the  same  length  as 
the  correct  answer? 


High 

Validity 

Yes  78.1# 
No  21.9 


Low 

Validity 

58.1# 

41.9 


Chi- 

Square 

14.545 


Level  of 
Significance 

.1# 


6.  Are  distracters  parallel  in  form  to  the  correct 


answer 


High 

Low 

Chi- 

Level  of 

Validity 

Validity 

Square 

Signif icance 

Yes 

No 

99.4# 

'•6 

87.5# 

12.5 

17.308 

.1# 
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. 


• i •'  ' 


' 


I. 


. 


. ' 

■ . r : 


. 

. 


T 


?*T. 


• - 


•'/ 


■ 


7. 

Are  distracters  parallel  in  meaning  to  the  correct 

answer? 

High 

Low 

Chi- 

Level  of 

Validity 

Validity 

Square 

Significance 

Yes  83.7$ 

70.6$ 

7.511 

1$ 

No  16.3 

29.4 

8. 

Do  the  distracters,  each  within 

themselves,  repeat 

conditions 

which  might 

have  been 

included  in  the 

premise? 

High 

Low 

Chi- 

Level  of 

Validity 

Validity 

Square 

Significance 

Yes  2.5$ 

8.1$ 

5.018 

5$ 

No  97.5 

91.9 

9. 

Is  the  statement  "None 

of  these 

answers"  or  "Correct 

answer  not 

given"  included  among 

the  distracters? 

Hi  gh 

Low 

Chi- 

Level  of 

Validity 

Validity 

Square 

Significance 

Yes  5.6$ 

7.5$ 

.370 

90$ 

No  94 . 4 

92.5 

10. 

Do  the  distracters  ( as 

well  as  the  correct  answers) 

consist  of 

single  ideas 

, or  do  they  each  present  a com- 

bination  of 

ideas  for  c 

omparison 

or  contrast? 

High 

Low 

Chi-  Level  of 

Validity  Validity  Square  Significance 

Simple  1^ 

68.2$ 

63.1$ 

.733  90$ 

Complex  2 

29.4 

34.4 

(No.  of  3 

1.3 

1.9 

ideas)  4(+)  1.3 

0.6 
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11 . Are  there  "specific  determiners"  Included  In  the 


dis tracters? 


High 

Validity 


Low  Chi-  Level  of 

Validity  Square  Signif icance 


Yes  0 
No  100 


0 % 


1.3# 

98.7 


1.568 


50# 


SUMMARY 

The  thirty- seven  questions  in  the  check  list  have  been 
evaluated  and  twenty  have  been  found  to  contain  points  on  which 
the  criterion  items  differed  significantly.  The  questions  to 
be  retained  for  the  further  test  of  effectiveness  described 
in  the  next  chapter  are  as  follows: 

The  Item  as  a Whole 

Question  No.  2,  3,  4,  5,  6,  8,  10,  11,  12 
The  Premise 

Question  No.  1,  2,  3,  4,**  5,  8 

The  Correct  Answer 
Question  No.  1 

The  Distracters 
Question  No.  4,  5,  6,  7,  8 

•^-Question  No.  4 retained  for  further  investigation  due  to 
inconclusive  evidence  in  the  evaluation. 
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CHAPTER  V 

EXPERIMENTAL  REVISION  OP  ITEMS 
OP  UNDETERMINED  VALIDITY 

I.  THE  DETAILS  OP  THE  EXPERIMENT 
A test  of  the  revised  check  list.  The  questions  remaining; 
in  the  check  list,  revised  according  to  the  findings  in  the 
preceding  chapter,  will  provide  the  basis  for  revision  of  the 
two  tests  described  in  Chapter  III.  The  comparison  of  the 
indices  of  validity  of  the  original  and  revised  forms  will  show 
the  effectiveness  of  the  revision  as  far  as  validity  is  con- 
cerned. The  significance  of  the  difference  between  mean  phi 
scores  will  be  determined  in  the  manner  already  indicated. 

Later  in  this  chapter  these  differences  will  be  shown  for  each 
of  the  two  tests  as  a whole  and  for  each  question  in  the 
revised  list  on  which  revisions  were  based.  Por  the  sake  of 
uniformity  and  ease  in  referring  to  the  check  list  questions, 
these  questions  in  the  revised  list  will  be  given  the  numbers 
by  which  they  were  identified  in  Chanters  II  and  IV. 

The  test  as  a whole.  The  mean  phi  index  of  original  items 
and  of  revised  items  in  each  test  will  be  computed  ( Table  5, 
Anpendix  C,  pp.PO^land  Table  8,  Appendix  D,  pp.142-3)  and  the 
significance  of  the  difference  determined.  In  each  test 
there  were  items  which  were  not  revised.  The  mean  phi  score 
will  be  computed  for  this  group  as  well. 
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Test  No,  1.  The  original  and  revised  forms  of  each  item 
in  this  test  are  shown  in  Appendix  C,  pp . 94-140  . The  data 
used  in  connection  with  the  computation  of  phi  for  each  item 
are  shown  in  Table  5 referred  to  previously.  The  revision  of 
this  test  is  summarized  as  follows: 


Name  of  Test- 
Level- 

Total  No.  of  items- 

Human  Relations 
College  Freshman 
55 

No.  of  items  revised-  38 
No.  of  items  unchanged-  17 


Type  of  item- 

4 responses 

Number  of  Forms 
IA 
IB 
IIA 
IIB 

Number  of  students  tested 
236 
176 
160 
156 

Test  No,  2.  The  original  and  revised  forms  of  each  item 
in  this  test  are  shown  in  Appendix  D,  pp.  145-204  . The  data 
used  in  the  computation  of  phi  for  each  itonare  shown  in 
Table  8 referred  to  previously.  The  revision  of  this  test  is 


summarized  as  follows: 

Name  of  Test- 
Level- 

Total  No.  of  Items- 

History  and  Civics 
Junior  High  School 
60 

No.  of  items  revised-  49 
No.  of  items  unchanged-  11 


Tvpe  of  Item- 

5 responses 

Number  of  Forms 
A 
B 

Number  of  students  tested 
140 
140 

Difficulties  in  revision. 

There  ere  several  difficulties 

encountered  in  revising  instructor-made  tests  in  a situation  in 

✓ 

which  the  revised  form  is  to  be  used  simultaneously  with  the 
original.  The  first  lies  in  the  fact  that  instructors  at  the 


. 


. 

. ; PC  i T ■ ‘ r rtf  ~rc 

in  ' : ->  i r ‘ 


r 

, 

c . 

ac 


. 

- c e s*ie  eJtri.i  ni 

fr  : ' r>  ' r r br  x 
.t  r ' <=-  ctf  bx' Tr  e 1 ' J ( 

• r,  ••.  : !'  " X.f 


*!r  . 

' c . 

-■  . O! 

. 

' ' v lot  ' ■ af  r i rr  ^ 

n ' >>  ' % - iJ  i o'  1 <i 

r : i'if  ■:■  ’ *’  ■ ■ . I ■ ' c 


college  freshman  level  find  it  very  unwise  to  repeat  an 
examination.  Each  test  is  for  the  most  part  a new  one  and  is 
prepared  only  a very  short  time  in  advance  of  the  date  of  its 
use.  There  is  hardly  time  for  adequate  revision.  The  second 
difficulty  occurs  when  the  work  of  revision  is  done  by  a third 
party,  one  who  is  not  a part  of  the  instructor-student 
situation.  An  item  which  may  appear  to  be  phrased  peculiarly 
to  one  not  in  close  rapport  with  the  instructors  and  students 
may  be  more  meaningful  to  the  students  than  one  whose  straight- 
forward wording  has  been  the  result  of  an  impersonal  revision. 

There  is  the  difficulty  in  reproducing  by  the  mimeograph 
process  comparable  forms  of  items  originally  set  in  type.  In 
Test  No.  2,  for  example,  the  responses  were  originally  printed 
horizontally  across  the  page  continuing  for  as  many  lines  as 
might  be  required.  This  plan  was  followed  as  closely  as 
possible  in  presenting  the  original  items.  The  responses  in 
the  revised  items  were  arranged  vertically.  For  the  sake  of 
uniformity,  the  responses  of  the  unchanged  items  were  arranged 
in  the  same  manner  as  those  of  the  rest  of  the  items  in  that 
part  of  the  test  in  which  they  occurred.  In  this  way  each 
unchanged  item  appeared  with  responses  arranged  both  horizon- 
tally and  vertically. 

An  insurmountable  problem  arises  when  a definite  and 

comprehensive  plan  of  revision  is  being  evaluated.  No  test  or 

✓ 

no  two  tests  designed  by  others  for  practical  use  will  provide 
opportunities  to  test  all  parts  of  the  plan  to  an  equal  extent. 
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Revision  should  be  made  only  when  items  are  found  to  need  it 
according  to  the  prescribed  pattern  in  the  check  list.  To 
attempt  to  change  an  item  for  the  sole  purpose  of  testing  some 
little  tested  point  in  the  list  when  that  item  already  is 
setisfactory  on  other  points  is  not  justifiable.  The  two  tests 
used  in  this  experiment  provided  no  opportunity  to  evaluate  to 
a satisfactory  extent  some  ooints  in  the  check  list. 


II.  THE  REVISION  AND  ITS  RESULTS 
The  revision.  In  Appendices  C and  D the  revision  of  each 
item  is  shown  together  with  a brief  explana  tion  of  the  changes 
made.  In  the  following  section  in  this  chapter  the  revised 
check  list  will  be  taken  up  point  by  point  and  the  items 
revised  in  accordance  with  esch  point  will  be  listed.  It  will 
be  seen  that  some  points  in  the  list  were  not  used  at  all  and 
that  others  were  used  very  infrequently. 

The  results  of  revision.  The  results  of  the  revision  of 
items  will  be  shown  below,  first  for  the  test  as  a whole,  and 
then  for  each  ooint  remaining  in  the  check  list. 

A.  The  Test  as  a Whole 

Level  of 

Original  Revised  Difference  t-score  Significance 

Test  No.l 

Mean  phi  .322  .342  .02  .09  None 

S.D.  .158  .156 

(See  Table  6,  Appendix  C,  p.  92  ) 


Test  No.  2 
Mean  phi 

.328 

.332 

.04 

.053 

None 

S.D. 

.179 
( See 

.208 

Table  9, 

Appendix  D, 

p.144  ) 
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Unchanged  Items 


Test  No.l 

Mean  phi  .271 

S.D.  .142 

Level  of 

Horizontal  Vertical  Difference  t-score  Significance 

Test  No. 2 

Mean  phi  .303  .330  .027  .120  None 

S.D.  .144  .173 


B.  The  Item  as  a Whole 

2 . Does  the  item,  combined  with  each  choice,  constitute 
a complete  unit  of  thought  grammatically? 

Items  Revised 
None 

3 . Does  the  premise,  combined  with  the  right  answer, 
constitute  a definite  and  true  concept? 

Items  Revised 
None 


4 .  Is  the  idea  stated  clearly  and  directly  with  the 
answer  an  important  part  of  the  statement? 

Items  Revised 


Test  No.l 

8,  10, 

11,  13, 

23,  25,  28 

Test  No. 2 

9,  10, 

11,  13, 

19,  21,  25,  29, 

46,  48, 

57 

Level  of 

Original 

Revised 

Difference  t- 

score 

Sierni  f icance 

Test  No.l 

Mean  phi 

S.D. 

.237 

.131 

.266 

.116 

.029 

.165 

None 

Test  No. 2 

Mean  phi 

S.D. 

.321 
.149  ' 

.412 

.169 

.091 

.405 

None 
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5 . Does  the  item  call  for  problem  solving,  for  interpreta- 
tion of  data,  or  for  the  recall  of  factual  information? 

Items  Revised 
None 

6 . Is  the  item  as  direct  a measure  as  possible  of  the 
points  of  mental  attainment  under  consideration? 

Items  Revised 

Test  No.  1 4 

Test  No.  2 3,  7,  23,  24,  26,  28,  32,  33,  34,  40,  47,  53,58,60 

Level  of 

Original  Revised  Difference  t-score  Significance 

Test  No.  1 

Mean  phi  .41  .46  .05 

S.D. 


Test  No.  2 

Mean  phi  .327  .297 

S.D.  .17  .22 


.03 


.388 


None 


8 . What  is  the  Index  of  Ease? 

Items  Revised 
None 

10 . Are  answers  arranged  horizontally  or  vertically? 

Items  Revised 


Test  No.  1 6,  12 

Test  No.  2 See  "Unchanged  Items"  Test  No.  2 


Level  of 

Original 

Test  No.  1 

Revised 

Difference 

t-score  Sls-nif  icance 

Mean  phi  .23 

S.D. 

30.5 

7.5 

1.042  None 
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11.  Is  the  item  self-contained  or  does  it  require  the  use 

- T 

of  reference  material  not  included  in  the  test? 

Items  Revised 
None 


12 . How  many  words  are  used  in  stating  the  item  as  a whole? 

Items  Revised 
None 


C.  The  Premise 

1 .  Is  a definite  task  clearly  and  unambiguously  set? 

Items  Revised 
None 


2 .  Is  the  wording  simple  and  direct? 

Items  Revised 

Test  No . 1 31 

Test  No.  2 55 

Level  of 

Original  Revised  Difference  t-score  Significance 

Test  No.  1 

Mean  phi  .70  .49  .21 

S.D. 

Test  No.  2 

Mean  phi  -.06  .00  .06 

S.D. 


3 .  Is  the  -premise  expressed  in  the  form  of  an  Incomplete 
statement  or  as  a question? 

Items  Revised 


Test  No.  1 2,  3,  5,  6,  9,  12,  13,  27,  34 
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Level  of 


Original 

Revised 

Difference 

t-score 

Significance 

Test  No 

. 1 

Mean 

£hl 

.330 

.381 

.051 

.279 

None 

S.D. 

.123 

.135 

4. 

Is 

the  statement  in 

the  premise 

positive 

or  negative? 

Items  Revised 


Test  No.  1 None 

Test  No.  2 5,  30 

Original  Revised 

Test  No.  2 

Mean  phi  .36  .335 

S.D.  .05  .215 

5 . Is  the  premise  so  phrased  that  the  "best  answer" 
rather  than  the  "correct  answer'1  is  called  for ? 

Items  Revised 
None 

8 . How  many  words  are  used  in  stating  the  premise? 

Items  Revised 
None 

D.  The  Correct  Answer 

1 . Is  there  only  one  correct  answer  to  satisfy  the 
premise,  an  answer  on  which  competent  authorities 
will  agree? 

Items  Revised 


Level  of 

Difference  t-score  Significance 


.025 


.113 


None 


. None 
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E.  The  Distracters 


4 . Are  distracters  too  close  to  correct  answer? 

Items  Revised 


Test  No.  1 None 

Test  No.  2 45 


Level  of 


Original  Revised  Difference 

Test  No.  2 

Mean  phi  .49  .68  .19 

S.D. 

5.  Are  distracters  of  approximately 

t-score 
the  same 

Significance 
length  as 

the  correct 

answer? 

Items 

Revised 

Test  No.  1 

e,  17, 

19,  23, 

26,  29,  32, 

35,  36,  37 

Test  No.  2 

42,  43 

Level  of 

Original 

Revised 

Difference 

t-score 

Significance 

Test  No.  1 

Mean  phi 

.366 

.344 

.022 

.137 

None 

S.D. 

.095 

.13 

Test  No.  2 

Mean  Phi 

.375 

.240 

None 

S.D. 

6.  Are  distracters  parallel  in  form 

to  the  correct  answer? 

Items 

Revised 

Test  No.  1 

15,  22, 

26,  28, 

29,  31,  35, 

36 

Test  No.  2 

2,  3, 

6,  10,  12 

, 14,  17,  20 

, 27,  29, 

30,  31,  42, 

43,  47, 

49,  50, 

51,  52,  59 

Level  of 

Original 

Revised 

Difference 

t-score 

Significance 

Test  No.  1 

Mean  phi 

.455 

.430 

.025 

.155 

None 

S.D. 

.109 

.118 

Test  No.  2 

✓ 

Mean  Dhi 

.323 

.350 

.027 

.101 

None 

S.D. 

.179 

.201 
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7 . Are  distracters  parallel  in  meaning  to  the  correct 


answer? 

Items  Revised 


Test  No.  1 

14,  15, 

16,  18, 

21,  22,  23,  24,  26, 

29,  30,  31, 

33,  35, 

36,  37, 

38 

Test  No.  2 

1,  2, 

o 
1 — 1 

iH 

•v 

tO 

’,  29,  30,  31,  35,  42 

, 43,  49,  50, 

51,  52, 

59 

Level  of 

Original 

Revised 

Difference  t-score 

Significance 

Test  No.  1 

Mean  phi 

.336 

.319 

.017  .068 

None 

S.D. 

.179 

.171 

Test  No.  2 

- 

Mean  phi 

.313 

.295 

.018  .065 

None 

S.D. 

.159 

.223 

8 . Do  the  distracters,  each  within  themselves,  repeat 
conditions  which  might  have  been  included  in  the 


•premise? 

Items  Revised 

Test  No.  1 1,  4,  7,  12,  13,  18,  20,  22,  24,  27,  32,  34, 

35,  36 

Test  No.  2 2,  6,  14,  17,  27,  29,  31,  35,  39,  49,  50 

Level  of 

Original  Revised  Difference  t-score  Significance 

Test  No.  1 


Mean  phi 

.348 

.362 

.014 

.079 

None 

S.D. 

.127 

.124 

Test  No.  2 

Mean  phi 

.371 

.402 

.031 

.134 

None 

S.D. 

.143 

.182 

S DM  MARY 


The  one  hundred  and  ten  multiple-choice  items  which  made 
up  the  two  tests  designed  for  practical  use  at  two  different 
levels  of  the  social  science  field  were  checked  on  each  point 
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in  the  revised  check  list.  All  items  were  found  satisfactory 
on  ten  of  these  twenty  points.  Revisions  were  made  in 
accordance  with  the  remaining  ten  points.  These  revisions 
have  produced  no  significant  differences  in  item  validity  both 
in  the  tests  as  a whole  and  in  the  itens  revised  according  to 
the  several  points  in  the  check  list. 
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CHAPTER  VI 

SUMMARY  aND  CONCLUSIONS 
I . SUMMARY 

The  purpose  of  this  study  was  to  determine  if  there  are 
certain  external  characteristics  which  distinguish  multiple- 
choice  items  of  high  validity  from  those  of  low  validity  and  to 
explore  the  possibility  of  formulating  rules  for  item  prepara- 
tion which  will  insure  the  incorporation  of  these  characteris- 
tics. In  Phase  I current  thought  and  practice  were  summarized 
in  a check  list  consisting  of  thirty-seven  suggestions  for  item- 
writing. A critical  evaluation  of  this  check  list  in  Phase  II 
showed  that  the  criterion  items  of  high  validity  differed  sig- 
nificantly from  those  of  low  validity  on  twenty  points.  In 
Phase  III  in  test  try-out  situations  on  two  different  educa- 
tional levels  it  was  found  that  items  revised  in  accordance 
with  these  twentv  points  showed  no  significant  change  in 
validity . 

II.  CONCLUSIONS 

The  conclusions  drawn  from  each  phase  follow  with  those 
derived  from  Phases  I and  II  presented  in  the  form  of  sugges- 
tions for  item  writing. 

A.  Present  Day  Guides  for  Item  W-riting 
✓ 

Suggestions  for  item  preparation  arising  from  the  summary 
of  current  thought  and  practice  are  8S  follows: 
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The  item  as  a whole. 


1.  The  premise,  combined  with  each  choice,  should 
constitute  a complete  unit  of  thought  ideologically. 

2.  The  premise,  combined  with  each  choice,  should  con- 
stitute a complete  unit  of  thought  grammatically. 

3.  The  premise,  combined  with  the  right  answer,  should 
constitute  a definite  and  true  concept. 

4.  The  idea  should  be  stated  clearlv  and  directly  with 
the  answer  an  important  part  of  the  statement. 

5.  The  use  of  multiple-choice  items  in  achievement  test- 
ing should  not  be  restricted  to  recall  of  factual 
information  but  may  be  extended  to  other  types  of 
items  such  as  those  which  require  problem  solving;  and 
the  interpretation  of  data. 

6.  The  item  should  be  as  direct  a measure  as  possible  of 
mental  attainment.  It  should  call  for  the  use  of  in- 
formation, whenever  possible,  rather  than  just  for  the 
information  itself. 

7.  Random  placing;  of  the  correct  choice  is  desirable. 

3.  Too  many  easy  or  too  many  difficult  items  in  a test 

are  not  desirable. 

9.  The  item  should  be  free  from  grammatical,  spelling, 
and  typographical  errors. 

10.  Answers  should  be  arranged  horizontally  whenever  the 
✓ 

nature  of  the  material  permits. 
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11.  The  use  of  reference  material  to  supplement  the  in- 
formation presented  in  the  premise  may  he  used. 

12.  The  item  should  he  worded  as  briefly  as  possible. 

The  premise . 

1.  A definite  task  should  be  clearly  and  unambiguously  set. 

2.  The  wording  should  be  simple  and  direct. 

3.  The  premise  may  be  expressed  either  in  the  form  of  an 
incomplete  statement  or  as  a question. 

4.  The  premise  may  be  stated  either  negatively  or 
positively . 

5.  Items  may  call  either  for  the  correct  answer  or  the 
best  answer. 

6.  The  item  should  be  based  on  a combination  or  a contrast 
of  ideas  rather  than  on  a single  idea. 

7.  The  nature  of  the  premise  may  be  such  that  other 
answers,  as  good  as  any  included  among  the  responses, 
require  the  use  of  "of  the  following". 

3.  The  premise  must  be  brief. 

The  correct  answer. 

1.  There  must  be  only  one  correct  answer  included  among 
the  responses  which  will  satisfy  the  premise,  an 
ansv/er  on  which  competent  authorities  will  agree. 

2.  The  answer  must  be  so  worded  that  it  cannot  be  deter- 
mined without  reference  to  the  premise. 

✓ 

3.  The  answer  must  be  free  from  distinguishing  character- 
istics such  as  length  or  repetition  of  words  in  the 
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premise . 

4.  The  responses  may  be  such  that  the  selection  of  the 
incorrect  answer  instead  of  the  correct  may  be  required. 

5.  The  correct  enswer  should  be  briefly  stated. 

6.  The  correct  answer  may  contain  a combination  or 
contrast  of  ideas. 

The  distracters. 

1.  All  distracters  should  be  plausible. 

2.  They  should  be  related  to  the  premise 

a.  ideologically 

b.  grammatically 

3.  Confusion  caused  by  using:  distracters  too  close  in 
meaning:  to  the  correct  answer  should  be  avoided. 

4.  The  length  of  all  responses  in  a given  item,  distrac- 
ters and  correct  answer,  should  be  apnroximately  the 
same . 

5.  The  distracters  should  be  parallel  to  the  correct 
answer  in 

a . form 

b.  meaning 

6.  They  should  not  repeat  within  themselves  conditions 
which  might  have  been  included  in  the  premise. 

7.  The  statement  "None  of  these  answers"  or  "Correct 
answer  not  given"  may  be  included  as  one  of  the 
distracters . 

8.  The  distracters  may  consist  of  single  ideas  or  they 

✓ 

each  may  present  a combination  of  ideas  for  comparison 
or  contrast. 
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9.  The  use  of  such  specific  determiners  as  "always1'  or 
"never"  should  he  avoided. 

B.  The  Evaluation  of  Present  Day  Guides 

In  drawing  conclusions  from  the  results  of  the  evaluation 
of  the  points  listed  above,  consideration  must  be  aiven  to 
those  susrerestions  for  item  preparation  on  which  no  difference 
was  found  as  well  as  to  those  which  the  study  shows  to  be 
statistically  si  cm  if icant . 

Check  list  points  logically  important  but  statistically 
insignificant . Seventeen  of  the  original  thirty-seven  points 
revealed  no  statistically  determined  significant  difference 
between  high  and  low  validity  items  and  are  consequently 
dropped  from  the  list  at  this  point.  It  must  be  pointed  out 
that  several  of  them  showed  no  difference,  not  because  they  are 
not  important  factors  in  an  item  but  because  the  editorial  care 
with  which  the  criterion  items  were  prepared  insured  their 
incorporation  in  all  items.  Check  list  points  which  common 
sense  indicates  are  essential  but  which,  because  of  the  limita- 
tions of  this  study  are  not  found  statistically  significant  are: 

The  item  as  a whole . 

1.  The  oremise,  combined  with  each  choice,  should  con- 
stitute a complete  unit  of  thought  ideologically. 

2.  The  item  should  be  free  from  errammatic  al , spelling, 
and  typo  err  aphi  cal  errors. 

s 

The  correct  answer . 

1.  The  answer  must  be  worded  so  that  it  cannot  be 
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determined  without  reference  to  the  premise. 

2.  The  answer  must  be  free  from  dis tinguishing  character- 
istics such  as  length  or  repetition  of  words  in  the 
pr  emi s e . 

The  distracters. 

1.  All  distracters  should  be  plausible. 

2.  Each  distracter  should  be  related  to  the  premise 
ideologically  and  grammatically. 

3.  The  use  .of  such  specific  determiners  as  "always"  or 
"never"  should  be  avoided. 

Statistically  significant  characteristics.  In  presenting 
the  suggestions  for  item  preparation  which  were  found  to  be 
significant,  the  points  will  be  arranged  in  a more  logical 
order  than  that  which  is  indicated  by  the  statistically  deter- 
mined degree  of  difference.  As  has  already  been  pointed  out 
this  degree  of  difference  between  items  of  high  and  low  validity 
has  been  greatly  affected  by  the  care  and  precision  with  which 
the  criterion  items  were  prepared.  Consequently  if  the  size  of 
chi-square  is  allowed  to  dictate  the  position  of  a point  in  the 
list,  a distorted  idea  of  the  relative  importance  of  the  points 
may  result.  Suggestions  for  item  preparation  based  on  the 
twenty  significant  points  follow  arranged  logically  in  the 
order  of  their  imnortance.  Their  position  on  a list  arranged 

according  to  the  degree  of  difference  which  was  found  to  exist 

✓ 

between  the  criterion  items  of  high  and  low  validity  is  indi- 
cated in  Darentheses. 
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The  item  as  a whole 


1.  (9)  The  premise,  combined  with  each  choice,  should 

constitute  a complete  unit  of  thought  grammatically. 

2.  (7)  The  premise,  combined  with  each  choice,  should 

constitute  a definite  and  true  concept. 

3.  (6)  The  idea  should  be  stated  clearly  and  directly  with 

the  answer  an  important  part  of  the  statement. 

4.  (8)  Items  which  involve  problem  solving  or  interpreta- 

tion of  data  show  higher  valid! ty  than  those  which 
involve  only  recall  of  factual  information. 

5.  (3)  The  item  should  call  for  the  use  of  information, 

whenever  possible,  rather  than  just  for  the  informa- 
tion itself. 

6.  (1)  The  easier  the  item,  the  less  valid  it  is. 

7.  (5)  Answers  should  be  arranged  horizontally  whenever 

the  nature  of  the  material  permits. 

8.  (4)  The  use  of  reference  material  to  supplement  the 

information  presented  in  the  premise  is  justifiable 
as  far  as  validity  is  concerned.  This  fact  opens 
up  many  testins  fields  to  the  use  of  multiple-choice 
items . 

9.  (2)  Clarity  in  expression  rather  than  brevity  is 

essential.  Brevity  should  be  determined  by  the 
nature  of  the  content. 

The  premise. 

1.  (1)  A definite  task  should  be  clearly  and  unambiguously 
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2.  (4)  The  wording  should  be  simple  and  direct. 

3.  (5)  Whenever  possible,  the  premise  should  be  stated  in 

the  form  of  an  incomplete  statement. 

4.  (3)  The  premise  should  be  so  phrased  that  the  correct 

answer  rather  than  the  best  answer  is  called  for. 

5.  (2)  Clarity  not  brevity  should  be  the  essential  con- 

sideration in  the  wording  of  an  item. 

The  correct  answer. 

1.  There  must  be  only  one  correct  answer  included 

among  the  responses  which  will  satisfy  the  premise, 
an  answer  on  which  competent  authorities  will  agree. 
The  distracters. 

1.  (2)  Confusion  caused  by  using  distracters  too  close  in 
meaning  to  the  correct  answer,  should  be  avoided. 

2.  (3)  The  length  of  all  responses  in  a given  item, 
distracters  together  with  the  correct  answer, 
should  be  approximately  the  same. 

3.  (1)  The  distracters  should  be  parallel  in  form  to 

the  correct  answer . 

4.  (4)  They  should  be  parallel  in  meaning  to  the  correct 

answer . 

5.  (5)  Phrases  which  might  have  been  included  in  the  pre- 

mise should  not  be  repeated  in  each  of  the  responses 
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C.  Experiments!  Revision  of  Items  of 
Undetermined  Validity 


The  fact  thst  the  revision  of  items  in  accordance  with  the 
twenty  points  enumerated  above  resulted  in  no  significant 
increase  in  validity  can  be  attributed  to  the  high  validity  of 
the  original  items,  to  the  editorial  care  with  which  they  had 
been  prepared,  and  to  the  lack  of  opportunity  they  presented 
for  adequate  experimentation  on  all  points  in  the  list.  In 
support  of  the  value  of  these  suggestion?  for  item  writing  it 
can  be  pointed  out  that  they  resulted  in  no  general  decrease  of 
validity  even  in  the  case  of  items  with  generally  high  dis- 
crimination indices. 


The  results  of  this  study  are  subject  to  the  following 

limitations : 

1.  The  items  which  made  up  the  criterion  groups  were 
based  on  subject-matter  scientific  and  technical  in 
nature.  There  may  be  some  relationship  between  the 
nature  of  this  subject-matter  end  the  characteristics 
of  items  which  proved  effective  in  the  examinations 
based  upon  it,  a relationship  different  from  that 
between  similar  items  and  social  science  material. 

2.  The  tests  selected  for  revision  did  not  provide 

adequate  opportunity  to  determine  to  an  equal  extent 
✓ 

the  value  of  all  points  in  the  check  list.  It  is  not 
probable  that  any  test  already  preoared  will  present 
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such  an  opportunity. 

3.  The  tests  selected  for  revision  showed  a high  degree 
of  item  validity.  The  items  had  been  so  well  prepared 
and  selected  that  there  was  small  probability  that 
revision  could  brine:  about  a significant  increase  in 
validity . 

IV.  POINTS  FOR  FURTHER  STUDY 

The  following  questions  for  further  research  are  suggested: 

1.  What  is  the  effect  on  validity  of  the  indiscriminate 
use  of  positive  and  negative  items  throughout  a test? 

2.  What  is  the  relationship  between  the  external 
characteristics  and  the  ease  of  an  item? 

3.  To  what  extent  can  the  scope  of  the  multiple-choice 
item  be  increased  by  the  use  of  supplementary 
materials  such  as  charts,  tables,  and  maps? 

4.  What  is  the  difference  in  the  validity  of  orisianl 
and  revised  items  in  a test  devised  specifically  to 
test  the  effectiveness  of  the  check  list? 

5.  What  is  the  effect  of  the  check  list  on  the  validity 
of  items  in  a test  revised  by  a writer  of  the 


original  test? 
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APPENDIX  B 


EVALUATION  OP  CRITERION  ITEMS 
TABULATED  DATA 


Table  1. 

Distribution  of  Phi  Coefficients  of  Test  Items 
in  AAP  Pilot  Ground  School  Achievement 
Testing  Program 


Phi 

f 

.66  - .70 

2 

.61  - .65 

2 

.56  - .60 

15 

.51  - .55 

36 

.46  - .50 

43 

.41  - .45 

82 

.36  - .40 

145 

.31  - .35 

186 

.26  - .30 

296 

.21  - .25 

327 

.16  - .20 

322 

.11  - .15 

262 

.06  - .10 

189 

.01  - .05 

90 

-.04  - .00 

45 

-.09  - -.05 

16 

-.14  - -.10 

12 

-.19  - -.15 

3 

-.24  - -.20 

1 

-.29  - -.25 

1 

N 

2075 

Mean  * .22 
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Table  2. 

Distribution  of  Phi  Coefficients  of 
Low  Validity  Criterion  Items 


phi 

f 

+.08  - +.10 

62 

+.05  - +.07 

34 

+.02  - +.04 

33 

-.01  - +.01 

15 

-.04  - -.02 

4 

-.07  - -.05 

4 

-.10  - -.08 

1 

-.13  - -.11 

5 

-.16  - -.14 

1 

-.19  - -.17 

0 

-.22  - -.20 

1 

N 

160 

Mean  » .045 

S.D.  = .057 



Table  3. 

Distribution  of  Phi  Coefficients  of 
High  Validity  Criterion  Items 


phi 

f 

.68  - .70 

2 

.65  - .67 

1 

.62  - .64 

1 

.59  - .61 

5 

.56  - .58 

7 

.53  - .55 

17 

. 50  - .52 

29 

.47  - .49 

23 

.44  - .46 

28 

.41  - .43 

47 

N 

160 

Mean  - .481 

<'  S.D.  — .053 

Table  4. 

Evaluation  of  Criterion  Groups 
Summary  of  Data 


High  , 

Low 

Chi- 

Sauare 

Level  of 
Sisinif . 

. Total/% 

Total/  % 

ITEM  AS  A WHOLE 

/ 

*1 . Ideological 

Yes 

160 

100.0 

160 

100.0 

0 

90 % 

unit 

No 

0 

0 

0 

0 

2.  Grammatical 

Yes 

160 

100.0 

154 

96.2 

5.776 

3% 

unit 

No 

0 

0 

6 

3.8 

3.  Definite  and 

Yes 

160 

0 

150 

93.7 

10.251 

1 % 

true  concept 

No 

0 

0 

10 

6.3 

4.  Stated  clearly 

Yes 

155 

96.9 

137 

85.6 

12.544 

. 1 % 

and  directly 

No 

5 

3.1 

23 

14.4 

5.  Problem 

P 

31 

19.5 

22 

13.8 

9.3 

1 % 

Int’pr’ t 'ns 

I 

87 

54.5 

59 

36.9 

Recall 

R 

42 

26.0 

79 

49.4 

6.  Direct  meas . 

Yes 

125 

78.0 

84 

52.5 

22.44 

. 1 % 

of  attainment 

No 

35 

22.0 

76 

47.5 

7.  Position  of 

a 

31 

19.4 

22 

13.8 

1.282 

90% 

correct 

b 

38 

23.8 

38 

23.8 

c 

36 

22.5 

37 

23.1 

d 

34 

21.3 

41 

25.6 

e 

21 

13.1 

22 

13.8 

8.  Index  of 

81- 

11 

6.9 

24 

15.0 

26.263 

. 1 % 

ease 

61-30 

63 

42.5 

54 

33.8 

41-60 

75 

46.9 

35 

21.9 

21-40 

6 

3.8 

32 

20.0 

-20 

0 

0 

15 

19.4 

9.  Grammatical 

Yes 

5 

3.2 

9 

5.6 

1.152 

30% 

errors 

No 

155 

96.8 

151 

94.4 

10.  Horizontal 

Hor . 

26 

16.3 

7 

4.4 

12.800 

. 1 % 

or  vertical 

Ver . 

134 

33.7 

153 

95.6 

11.  Item  self- 

Yes 

56 

35.0 

92 

57.5 

16 .340 

. 1 % 

contained 

No 

104 

65.0 

68 

42.5 

12.  Number 

56- 

77 

43.1 

28 

17.5 

25.08 

.1% 

of  words 

46-55 

5 

3.2 

23 

14.4 

36-45 

12 

7.5 

48 

30.0 

26-35 

24 

15.0 

31 

19.4 

-25 

42 

26.3 

30 

13.8 

*These  numbers  are  the 

same  as  the  numbers 

of  the  corresponding 

paragraphs  in  Chanter  II  and  Chapter  IV 
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Table  4.  (continued) 


Hi^h 

Low 

Chi- 

Square 

jevel  of 
Siffnif . 

Total/  % 

Total  / % 

THE  PREMISE 

7 

7 

1.  Stated 

Yes 

153 

95.6 

128 

80.0 

17.689 

. 1 % 

clearly 

No 

7 

4.4 

32 

20.0 

2.  Expressed 

Yes 

157 

98.1 

146 

91.2 

7.840 

1 % 

simply 

No 

3 

1.9 

14 

8.8 

3.  Statement 

S 

139 

86.9 

125 

78.1 

4.425 

5% 

or  question 

0. 

21 

13.1 

35 

21.9 

4.  Positive  or 

p 

160 

100.0 

158 

98.7 

1.568 

50% 

negative 

N 

0 

0 

2 

1.3 

5.  Best  or 

B 

8 

5.0 

19 

11.9 

5.284 

5 % 

correct 

C 

152 

95.0 

141 

88.1 

6.  Simple  or 

s 

35 

21.9 

48 

30.0 

2.486 

50% 

complex 

C2 

23 

14.4 

29 

18.1 

C3 

13 

8.1 

13 

8.1 

C4 

89 

55.6 

70 

43.8 

7.  "Of  the 

Yes 

8 

5.0 

11 

6.9 

.533 

50 % 

following'' 

No 

152 

95.0 

149 

93.1 

8.  Number  of 

21- 

86 

53.8 

52 

32.5 

10.677 

5% 

words 

17-20 

8 

5.0 

24 

15.0 

13-16 

20 

12.5 

35 

21.9 

9-12 

21 

13.1 

26 

16.3 

- 8 

25 

15.6 

23 

14.4 

CORRECT  ANSWER 

1.  One  correct 

Yes 

159 

99.4 

146 

91.2 

10.758 

1% 

answer 

No 

1 

.6 

14 

8.8 

2.  Obvious  word 

Yes 

2 

1.3 

3 

1.9 

.512 

50% 

clues 

No 

158 

98.7 

157 

98.1 

3.  Other  char- 

Yes 

5 

3.2 

7 

4.4 

.128 

50% 

acteris tics 

No 

155 

96.8 

153 

95.6 

4.  Correct  or  in- 

C 

159 

99.4 

158 

98.7 

.512 

50 % 

correct  answer 

I 

1 

.6 

2 

1.3 

5.  Number  of  words  11- 

12 

7.5 

15 

9.4 

.812 

95 % 

9-10 

10 

6.3 

9 

5.6 

7-8 

10 

6.3 

15 

9.4 

5-6 

22 

13.8 

22 

13.8 

-4 

106 

66 . 3 

99 

61.9 

6 . Simple  or 

s 

116 

72.5 

116 

72.5 

.267 

00% 

complex 

C2 

40 

25.0 

40 

25.0 

C3 

2 

1.3 

3 

1.9 

C4 

2 

1.3 

1 

.6 
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Table  4.  (concluded) 


Hish 

A Low 

Chi- 

jevel  of 

Total/  % 

YTota]/  $ 

. Square 

ISiynif . 

DISTRACTERS 
1.  Are  all 

Yes 

132 

82.5 

134 

83.8 

.096 

95$ 

claus ible 

No 

28 

17.5 

26 

16.2 

99$ 

2.  Related 

Yes 

160 

100.0 

160 

100.0 

0 

ideologically 

No 

0 

0 

0 

0 

50$ 

3.  Related 

Yes 

158 

98.7 

159 

99.4 

.512 

srammatic  ally 

No 

2 

1.3 

1 

.6 

.1$ 

4.  Too  close  to  - 

Yes 

6 

3.8 

27 

16.9 

15.022 

correct  answer 

No 

154 

96.2 

133 

83.1 

.1$ 

5.  Same  length 

Yes 

125 

78.1 

93 

58.1 

14.545 

as  correct 

No 

35 

21.9 

67 

41.9 

6.  Parallel  in 

Yes 

159 

99.4 

140 

87.5 

17.308 

.1$ 

form 

No 

1 

.6 

20 

12.5 

1$ 

7.  Parallel  in 

Yes 

134 

83.7 

113 

70.6 

7.511 

meaning 

No 

26 

16.3 

47 

29.4 

5$ 

8.  Repetition 

Yes 

4 

2.5 

13 

8.1 

5.018 

of  phrases 

No 

156 

97.5 

147 

91.9 

60$ 

9 . "None  of 

Yes 

9 

5.6 

12 

7.5 

.370 

these" 

No 

151 

94.4 

148 

92.5 

90$ 

10.  Simple  or 

S 

109 

68.2 

101 

63.1 

.733 

complex 

C2 

47 

29.4 

55 

34.4 

C3 

2 

1.3 

3 

1.9 

C4 

2 

1.3 

1 

.6 

11.  "Specific 

Yes 

0 

0 

2 

1.3 

1.568 

30$ 

determiners" 

No 

160 

100 

158 

98.7 
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APPENDIX  C 


TEST  NO . 1 


tabulated  data 

ORIGINAL  AND  REVISED  ITEMS 


Table  5. 

Phi  Coefficients  of  Validity 
Original  and  Revised  Items 
Test  No.  1 


Original  Form  Revised  Form 


Iteir 

Number* 

% 

% 

phi 

Number 

% 

% 

Phi 

No. 

in  each 

upper 

lower 

in  each 

upper 

lower 

criterion 

group 

group 

criterion 

group 

group 

group 

passing 

passing 

group 

passing 

passing 

1 

59 

81 

59 

.25 

44 

95 

34 

.65 

2 

59 

82 

36 

.46 

44 

84 

23 

.60 

3 

59 

-81 

61 

.30 

44 

98 

52 

.53 

4 

138 

79 

38 

.41 

44 

91 

48 

.46 

5 

59 

80 

48 

.32 

44 

77 

39 

.38 

6 

59 

82 

59 

.25 

59 

91 

59 

.37 

7 

59 

36 

15 

.26 

44 

41 

11 

.35 

8 

59 

97 

76 

.30 

44 

100 

75 

.35 

9 

59 

83 

59 

.27 

44 

72 

41 

.31 

10 

99 

47 

34 

.13 

83 

54 

24 

.30 

11 

59 

93 

88 

.10 

44 

98 

84 

.26 

12 

39 

92 

77 

.21 

. 40 

85 

63 

.24 

13 

59 

56 

49 

.07 

44 

61 

43 

.81 

14 

59 

31 

46 

- .16 

44 

55 

50 

.05 

15 

59 

74 

22 

.52 

44 

91 

41 

.52 

16 

59 

85 

34 

.50 

44 

87 

36 

.51 

17 

59 

98 

71 

.38 

44 

98 

73 

.37 

18 

59 

58 

27 

.31 

44 

57 

25 

.32 

19 

59 

73 

51 

.23 

44 

96 

84 

.22 

20 

59 

73 

54 

.20 

44 

66 

45 

.21 

21 

59 

97 

66 

.40 

44 

100 

77 

.35 

22 

39 

93 

46 

.50 

40 

78 

52 

.27 

23 

138 

81 

38 

.42 

44 

66 

43 

.23 

24 

59 

95 

63 

.39 

44 

100 

82 

.32 

25 

59 

85 

66 

.23 

44 

84 

82 

.07 

26 

138 

82 

38 

.44 

44 

87 

25 

.61 

27 

39 

62 

26 

.37 

40 

80 

25 

.54 

28 

59 

75 

36 

.39 

44 

100 

61 

.47 

29 

59 

83 

59 

.27 

44 

91 

61 

.35 

30 

39 

95 

85 

.19 

40 

95 

80 

.25 

-*See  Table  7,  n.  93 
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Table  5.  (cone 

luded) 

Original  Form 

Revised  Form 

Item 

Number 

% 

% 

Phi 

Number 

% 

% 

Phi 

No. 

in  each 

uoper 

lower 

in  each 

upper 

lower 

criterion 

group 

group 

criterion  group 

group 

group 

passing 

passing 

group 

passing 

passing 

31 

59 

93 

31 

.70 

44 

95 

50 

.49 

32 

39 

90 

39 

.51 

40 

95 

55 

.46 

33 

59 

-92 

44 

.50 

44 

66 

55 

.17 

34 

40 

98 

55 

.50 

39 

90 

62 

.32 

35 

40 

100 

60 

.47 

39 

93 

56 

.43 

36 

40 

78 

40 

.38 

39 

77 

49 

.28 

37 

39 

97 

85 

.21 

40 

93 

85 

.15 

38 

138 

29 

17 

.15 

44 

20 

23  - 

.03 

39* 

103 

69 

21 

.48 

40 

103 

32 

33 

.48 

41 

79 

100 

68 

.42 

42 

103 

93 

58 

.40 

43 

103 

96 

65 

.33 

44 

182 

82 

49 

.33 

45 

103 

100 

80 

.32 

46 

103 

61 

28 

.32 

47 

103 

58 

28 

.30 

48 

103 

69 

45 

.25 

49 

79 

88 

70 

.24 

50 

142 

77 

57 

.23 

51 

103 

97 

86 

.22 

52 

123 

52 

33 

.19 

53 

103 

71 

59 

.13 

54 

103 

53 

43 

.10 

55 

103 

11 

20 

-.14 

*Items  No.  39  to  55  unchanged 
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Table  6 . 

Distribution  of  Phi  Coefficients  of  Validity 
of  Test  Items  in  Test  No.  1 


Phi  \ Original 

Revised 

Unchanged 

\ 

Form 

Form 

. . \ 

f 

f 

f 

.66  - -.70 

1 

0 

0 

.61  - .65 

0 

2 

0 

.56  - .60 

0 

1 

0 

.51  - .55 

2 

4 

0 

.46  - .50 

6 

4 

2 

.41  - .45 

3 

1 

1 

.36  - .40 

6 

4 

2 

.31  - .35 

2 

7 

3 

. 26  - .30 

5 

4 

1 

.21  - .25 

6 

5 

4 

.16  - .20 

2 

2 

1 

.11  - .15 

2 

1 

1 

.06  - .10 

2 

1 

1 

.01  - .05 

0 

1 

0 

-.04  - .00 

0 

1 

0 

-.09  --.05 

0 

0 

0 

-.14  --.10 

0 

0 

1 

-.19  --.15 

1 

0 

0 

N 

38 

38 

17 

Mean 

.322 

.342 

.271 

S.D 

• 

.158 

.156 

.142 
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Table  7. 

Significant  Phi  Values  for  Values  of  N 
Test  No.  1 


N* 

5# 

it 

•i* 

r stst 

Y 6 .64 

rTSTSjr 

I N 

1 N 

l N 

35 

.23 

.31 

.39 

39 

.22 

.29 

.37 

40 

.22 

.29 

.37 

44 

.21 

.28 

.35 

59 

.18 

.23 

.30 

79 

.15 

.21 

.26 

83 

.15 

.20 

.26 

99 

.14 

.13 

.23 

103 

.13 

.18 

.23 

125 

.13 

.17 

.21 

138 

.12 

.16 

.20 

142 

.12 

.16 

.20 

182 

.10 

.14 

.17 

*N  = the  number  in  each  criterion  croup. 
In  the  computation  of  phi  these 
values  of  N must  be  doubled. 
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Item  No.  1 


N Pu  Pi  0 


Original 


59  81  59  .95 


Judaism  tesches  its  followers  to 

(a)  abhor  strangers. 

(b)  ignore .strangers . 

(c)  distrust  strangers. 

-*(d)  treat  strangers  well. 


Strangers,  according  to  the  teachings  of  Judaism,  are  to  be 

(a)  abhorred. 

(b)  ignored. 

(c)  distrusted. 

*(d)  befriended. 

Revision . Distracters  have  been  simplified  by  inclusion 
of  "strangers"  in  premise. 


Revised 


44  95  34  .63 


. o!/i 


. 

. rr 

’ ' . 

. f,  (k.) 
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. " ' • . 

n i«!  ' is-  | 

. 
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Item  No*  2 


N Pu  Pi  0 


Original 


59  32  36  .46 


What  is  the  psychological  reason  for  "tabloid  thinking" 
quoted  from  Thouless  by  the  "ABC's"? 

-*(a)  Mental  idleness. 


The  psychological  reason  for  "tabloid  thinking",  as  quoted 
from  Thouless  by  the  "ABC's",  is 

*(a)  mental  idleness. 

(b)  desire  to  read  newspapers. 

(c)  fear  of  public  disapproval.  v 

(d)  None  of  the  above. 


(b)  Desire  to  read  newspapers. 

(c)  Pear  of  public  disapproval. 

(d)  None  of  the  above. 


Revised 


44  84  23  .60 


Revision . 


Premise  has  been  restated  in  statement  form. 
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Item  No.  3 


N Pu  Pi  t 


Original 


59  8B  61  .30 


Which  of  the  following  proclaimed  the  Italians  to  be 
pure  Aryan? 

(a)  Ripley. 

(b)  Gobineau. 

(c)  H.  S.  Chamberlain. 

*(d)  None  of  the  above. 


The  claim  that  the  Italians  were  pure  Aryan  was  made  by 

(a)  Ripley. 

(b)  Gobineau. 

(c)  H.  S.  Chamberlain. 

*(d)  None  of  the  above. 


Revised 


44  98  52  .53 


Revision 


Premise  has  been  restated  in  statement  form. 
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Item  No.  4 N Pu  Pi  $ 

Original  138  79  38  .41 

The  physical  anthropologists  define  "stock"  as 

(a)  one  of  the  three  major  genes  of  homo-sapiens . 

(b)  one  of .the  three  major  species  of  homo-sapiens. 
-*(c)  one  of  the  three  major  varieties  of  homo-sapiens. 

(d)  None  of  the  above. 

Revised  44  91  48  .46 

The  physical  anthropologists  would  classify  "Nordic" 
as  one  of  the  three  major 

(a)  genes  of  homo-sapiens. 

(b)  species  of  homo-sapiens. 

-*(c)  varieties  of  homo-sapiens. 

(d)  genes  of  homo-neanderthal. 

Revision.  1.  Premise  has  been  restated  to  include  phrases 
from  the  distracters. 

2.  Application  of  knowledge  is  called  for. 
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Item  No.  5 


N Pu  Pi  0 


Original 


59  30  48  .32 


Which  of  the  following  is  not  a motive  for  scapegoating, 
as  stated  in  '"ABC’s"? 

(a)  Guilt. 

(b)  Conformity. 

-*(  c ) Inner  Security. 

(d)  Self- enhancement . 


Prom  the  following  list  of  motives  select  the  one  which 
is  not  given  as  a motive  for  scapegoating  in  "ABC's". 

(a)  Guilt. 


Revised 


44  77  39  .33 


(b)  Conformity. 

*(c)  Inner  Security, 
(d)  Self- enhancement . 


Revision 


Premise  has  been  restated  in  statement  form 
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Item  No.  6 


K Pu  ?!  0 


Original 


59  82  59  .25 


Which  one  of  the  following  is  the  most  friendly  social 
relationship,  according  to  "ABC's*1? 

(a)  Respect. 

(b)  Tolerance. 

(c)  Preference. 

■K-(d)  Cooperation. 


The  most  friendly  of  the  following  social  relationships, 
according  to  "ABC’s",  is 

(a)  respect,  (b)  tolerance,  (c)  preference .*( d)  cooperation. 

Revision . 1.  Premise  has  been  rephrased  in  statement  form. 

2.  Responses  have  been  arranged  horizontally. 


Revised 


59  91  59  .57 
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Item  No.  7 


N Pu  Pi  0 


Original 


59  36  15  .26 


Parkes  feels  that  laws  passed  outlawing  antisemitism  are 

(a)  utterly  worthless. 

(b)  useful  -if  strict  enough. 

*(c)  useful  only  if  part  of  a larger  campaign. 

(d)  of  little  value  until  attitudes  are  changed. 


Parkes  feels  that  laws  passed  outlawing  antisemitism 
are  useful  only 

(a)  to  a small  degree. 

(b)  if  strict  enough. 

■*( c ) if  part  of  a larger  campaign. 

(d)  if  attitudes  are  first  changed. 

Revision . Repetition  of  phrases  in  distracters  has  been 


Revised 


44  41  11  .35 


avoided 
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Item  No.  8 


N Pu  Pi  0 


Original 


59  97  76  .30 


Which  of  the  following  statements  best  characterizes  the 
population  movement  of  the  Negro  since  1910? 

-*(a)  Negroes  migrated  to  Northern  cities. 

(b)  Negroes  migrated  mainly  toward  Southern  cities. 

(c)  Negroes  migrated  to  the  wide  oren  farm  lands  of  the 
Mid-West. 

(d)  Negroes  concentrated  in  the  rural  areas  of  the 
North  and  South. 


Which  of  the  following  statements  best  characterizes  the 
population  movement  of  the  Negro  since  1910? 

*(a)  Negroes  migrated  to  Northern  cities. 

(b)  Negroes  migrated  toward  Southern  cities. 

(c)  Negroes  migrated  to  the  farm  lands  of  the  Mid-West. 

(d)  Negroes  concentrated  in  the  rural  areas  of  the  North 


Revision . 1.  Unnecessary  adjectives  and  adverbs  have  been 

omitted . 

2.  Dlstracters  are  more  nearly  uniform  in  length. 


Revised 


44  100  75  .35 
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Item  No . 9 N Pu  P]_  0 

Original  59  33  59  .27 

Our  discrimination  against  the  Japanes e-Americans  in 
World  War  II  represents  which  motive  of  the  following? 


(a) 

Guilt . 

(b) 

Conformity. 

(c) 

Tabloid  Thinking. 

*(  d ) 

Fear  and  Anxiety. 

Revised  44  72  41  .31 

Our  motive  for  discrimination  against  the 
Japanese-Americans  in  World  War  II  was  thst  of 


(a) 

guilt . 

(b) 

conformity . 

(c) 

tabloid  thinking. 

*(d) 

fear  and  anxiety. 

Revision . 


Premise  has  been  rephrased  in  statement  form. 
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Item  No.  10 

N 

Pu 

Pi 

0 

Original 

99 

47 

34 

.13 

The  concept 

of  "race" 

originated  in 

(a)  the  Roman  Empire. 

(b)  Medieval  Times. 

*(c)  The  Renaissance. 

(d)  the  Nineteenth  century. 

Revised  83  54  24  .30 

The  concept  of  "race"  originated  during 

(a)  the  Roman  Empire. 

(b)  Medieval  times. 

•*(c)  the  Renaissance. 

(d)  the  Nineteenth  century. 

Revision.  Statement  has  been  clarified. 
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Item  No.  11 


N Pu  Pi  <t> 


Original 


59  93  88  .10 


"Taxonomy m may  be  defined  as 

(a)  the  means  of  differentiating  races. 

■fr(b)  the  science  of  biological  classification. 

(c)  the  science  of  taxation  and  public  finance. 

(d)  None  of  the  above. 


"Taxonomy"  is  defined  as 

(a)  the  means  of  differentiating  races. 

*(b)  the  science  of  biological  classification. 

(c)  the  science  of  taxation  and  public  finance. 

(d)  None  of  the  above. 

Revision.  Statement  has  been  made  definite. 


Revised 


44  98  84  .26 
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Item  No.  12  N Pu  Pi  0 

Original  39  92  77  .21 

Which  of  the  following  measures  the  configuration  of  the  jaw? 
( a)  Nose  index . 

*-(b)  Gnatic  index. 

(c)  Cranial  index. 

(d)  Cephalic  index. 

Revised  40  85  63  .24 

Select  from  the  following  list  that  index  which 
measures  the  configuration  of  the  jaw. 

(a)  Nose.  (b)  Cranial,  *(c)  Gnatic.  (d)  Cephalic. 

Revision . 1.  "Index"  has  been  included  in  premise. 

2.  Answers  have  been  arranged  horizontally. 

3.  Premise  has  been  rephrased  in  statement  form. 
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Item  No.  13 

N 

Pu 

Pi 

4 

Original 

59 

56 

49 

.07 

Which  of 
The  Races 

the  following  expressed 
of  Europe? 

the 

thesis 

of 

Ripley  in 

(a)  It  reinforced  the  Nordic  myth. 

(b)  It  refuted  the  arguments  of  Gobineau. 

*-(c)  It  demolished  the  theory  of  an  Aryan  race. 

(d)  It  emphasized  the  historical  reasons  for 
anti-Semitism. 

Revised  44  61  43  .18 

The  thesis  of  Ripley,  as  stated  in  The  Races  of  Europe, 

(a)  reinforces  the  Nordic  myth. 

(b)  refutes  the  arguments  of  Gobineau. 

-*(  c ) demolishes  the  theory  of  an  Aryan  race. 

(d)  emphasizes  the  historical  reasons  for  anti-Semitism. 

Revision . 


1.  Statement  has  been  clarified. 

2.  Premise  has  been  rephrased  in  statement  form. 

3.  Repetition  of  "it"  has  been  avoided. 
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Item  No.  14 

N 

Pu 

Pl  0 

Original 

59 

31 

46  -.16 

The  first 

manifestations 

of  conflict 

in  an 

individual 

arise  from  the 

(a)  reading  of  newspapers. 

*(b)  personal  social  relations. 

(c)  maturation  of  innate  prejudice. 

(d)  culturally  defined  relationships. 

Revised  44  55  50  .05 

The  first  manifestations  of  conflict  in  an  individual 
arise  from 

(a)  political-economic  relationships. 

-*(b)  personal  social  relations. 

(c)  maturation  of  innate  prejudice. 

(d)  culturallv  defined  relationships. 

Revision. 


Distrecters  have  been  made  parallel  in  meaning. 
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Item  No.  15 


N Pu  Pi  0 


Original 


59  74  22  .52 


Parkes  believes  that  the  Palestine  issue  should  be 
decided  in  favor  of  the  Jews  because 

*•(  a)  the  need  is  greater  and  would  cause  less  hardship 
than  if  decided  for  the  Arabs. 

( b)  the  Jews  served  honorably  and  courageously  during 
World  War  II  on  the  side  of  the  United  Nations 
whereas  the  Arabs  sided  with  the  Axis. 

(c)  viewed  historically,  the  claims  of  the  Jews  antedate 
the  claims  of  the  Arabs  by  many  hundreds  of  years. 

(d)  the  Jews  have  demonstrated  over  a short  period  of 
years  their  ability  to  transform  Palestine  into  a 
fertile  and  productive  area. 


Parkes  believes  that  the  Palestine  issue  should  be 
decided  in  favor  of  the  Jews  because 

*( a)  their  need  is  greater  and  they  would  suffer  more 
than  the  Arabs  from  an  adverse  decision. 

(b)  their  service  during  World  V/ar  II  was  on  the  side 
of  the  United  Nations  while  the  Ar*abs  served 
with  the  Axis. 

(c)  their  claims,  historically,  antedate  those  of  the 
Arabs  by  many  hundreds  of  years. 

(d)  their  ability  to  transform  the  country  into  a fertile 
and  productive  area  has  been  demonstrated. 


Revised 


44  91  41  .52 


Revision 


Distracters  have  been  made  parallel  in  form 
and  meaning. 
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Item  No.  16  N Pu  Pi 

Original  59  95  34 

Souls  of  Black  Folk  depict  the  ne^ro  as  a 

(a)  comic  figure. 

-'Kb)  tragic  figure. 

(c)  exotic  figure. 

(d)  contented  slave. 

Revised  44  87  36 

In  Souls  of  Black  Folk  the  negro  is  depicted  as 
(a)  comic  figure. 

■*Kb)  tragic  figure. 

(c)  exotic  figure. 

(d)  contented  figure. 
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Revision. 


Distracters  have  been  made  parallel  in  meaning. 
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Item  No.  17 


N Pu  Pi  7> 


Original 


59  98  71  .38 


Almost  the  only  flaw  in  the  Nazi  system  of  antisemitism 
was  the 

(a)  steadfast  resistance  of  the  Christian  church  to 
this  propoggnda. 

(b)  role  of  the  Prussian  military  clique  whose  members 
despised  Hitler. 

-*(c)  impossibility  scientifically  to  establish  what 
exactly  it  was  to  be  a Jew. 

(d)  refusal  of  the  universities  to  give  more  than  lip 
service  to  the  known  lies  the  Nazis  were  spreading 
about  the  Jews. 


Almost  the  only  flaw  in  the  Nazi  system  of  antisemitism 
was  the 

( a)  steadfast  resistance  of  the  Christian  church  to 
this  propaganda. 

(b)  role  of  the  Prussian  military  clique  whose  members 
despised  Hitler. 

-*(c)  impossibility  scientifically  to  establish  what 
exactly  it  was  to  be  a Jew. 

(d)  refusal  of  the  universities  to  accept  the  Nazi  lies 
about  the  Jews. 


Revised 


44  98  73  .37 


Revision . 


Distracters  have  been  made  uniform  in  length 
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Item  No.  18  N Pu  Pi  $ 

Original  59  58  27  .31 

The  best  reason  for  explaining:  the  rise  of  antisemitism 
in  Europe  in  the  latter  part  of  the  nineteenth  century  is 
that  the 

(a)  liberals  used  it  as  a means  to  gain  office. 

(b)  nationalists  employed  it  to  bring  unity  to  their 
countries . 

■*(c)  conservatives  utilized  it  to  secure  their  political 
control . 

(d)  Jews  controlled  political  power  through  the 
House  of  Rothschild. 

Revised  44  57  25  .32 

The  best  reason  for  explaining  the  rise  of  antisemitism 
in  Europe  in  the  latter  part  of  the  nineteenth  century 
is  its  use  by  the 

(a)  liberals  to  gain  office. 

(b)  nationalists  to  bring  unity  to  their  countries. 

*(c)  conservatives  to  secure  their  political  control. 

(d)  bankers  to  gain  control  of  the  House  of  Rothschild. 


Revision.  1.  Distracters  have  been  made  parallel  in  meaning. 

2.  Statement  h.8S  been  reworded  to  avoid 

unnecessary  repetition  of  phrases  in  dis- 
tracters . 
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Item  No.  19 


N Pu  Pl  0 


Original 


59  73  51  .23 


Which  of  the  following  expresses  Dr.  Eddy’s 
opinion  on  racial  interbreeding? 

■*(  a)  It  produces  no  harmful  effects. 

(b)  It  produces  innumerable  race  riots. 

(c)  It  results  ultimately  in  decreased  vitality. 

(d)  It  produces  many  neurotic,  unstable  personalities. 


Which  of  the  following  expresses  Dr.  Eddy's  opinion 
on  racial  interbreedine? 

*( a)  It  produces  no  harmful  effects. 

(b)  It  produces  innumerable  race  riots. 

(c)  It  produces  decreased  vitality. 

(d)  It  produces  neurotic,  unstable  personalities. 


Revised 


44  96  84  .22 


Revision.  Distracters  have  been  made  uniform  in  lensrth 
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Item  No.  20 


N Pu  Pi  0 


Original 


59  73  54  .20 


Mediterranean  and  Occidental  people  in  comparison 
with  the  Americans  possess 

(a)  no  Negro  prejudice. 

*(b)  less  Negro  prejudice. 

(c)  greater  Negro  prejudice. 

(d)  increasing  Negro  prejudice. 


Mediterranean  snd  Occidental  people,  in  comparison  with 
the  Americans,  possess  Negro  prejudice 

(a)  to  no  degree . 

*(b)  to  a smaller  degree. 

(c)  to  a greater  degree. 

(d)  to  an  increasing  degree. 

Revision.  Premise  has  been  reworded  to  avoid  repetition 
of  phrases  in  distracters. 


Revised 


44  66  45  .21 
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Item  No.  21 


N Pu  Pi  0 


Original 


59  97  66  .40 


According  to  Young,  one  of  the  main  reasons  for 
Alfred  Smith's  defeat  in  1928  was  the  fact  that  he 

(a)  championed  the  Negro  cause. 

(b)  was  an  incessant  drinker. 

(c)  held  a prejudice  against  foreigners. 

-*(d)  was  a member  of  the  Catholic  faith. 


According  to  Young,  one  of  the  main  reasons  for 
Alfred  Smith’s  defeat  in  1928  was  the  fact  that  he 

(a)  championed  the  Negro  cause. 

(b)  was  an  active  Democrat. 

(c)  held  a prejudice  against  foreigners. 

*(d)  was  a member  of  the  Catholic  faith. 

Revision.  Distracters  have  been  made  parallel  in  meaning. 


Revised 


44  100  77  .35 
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Item  No.  22  N Pu  Pi  0 

Original  39  93  46  .50 

Woodward  and  Sutherland  believe  that  racial  prejudice  is 

( a)  innate . 

(b)  a product  of  heredity. 

-*(c)  a result  of  social  experience. 

(d)  a combination  of  inherited  and  environmental  factors. 

Revised  40  79  52  .27 

Woodward  and  Sutherland  believe  that  racial  prejudice  is 
the  result  of 

(a)  heredity. 

(b)  physical  environment. 

-*(  c ) social  experience. 

(d)  political  expediency. 

Revision.  1. Statement  has  been  reworded  to  simplify 
distrac ters . 

2 . Distrac ters  have  been  made  parallel  to  correct 
answer  in  form  and  meaning. 

3.  Length  of  distracters  has  been  made  uniform. 
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Item  No. 

23  N Pu  Pi  jZf 

Original  138  81  38  .42 

"Race" 

developed  in  the  United  States  when  the  whites 

(a) 

wished  to  exploit  the  Jews. 

*(  b) 

wished  to  exploit  the  Negro. 

(c) 

sought  to  exploit  the  Indians. 

(d)  desired  to  portray  the  superiority  of  American 
culture . 

Revised  44  66  43  .23 

The  concept  of  "race"  developed  in  the  United  States 
when  the  whites  sought  to  exploit 


(a) 

the  Jews. 

*(b) 

the  Neero. 

(c) 

the  Indians. 

(d) 

cheap  immigrant  labor. 

Revision . 1.  Statement  has  been  clarified  by  "race". 


2.  Distracters  have  been  made  uniform  in 
length  and  parallel  in  meaning. 
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Item  No.  24  N Pu  P^  $ 

Origins!  59  95  63  .39 

The  main  use  of  classification  of  races  is 

(a)  to  prove  the  existence  of  biological  races. 

(b)  to  thwart  the  doctrines  of  race  propoganda. 

(c)  to  discover  which  race  has  the  best  potentialities. 

*(d)  to  serve  as  a statistical  concept  for  the 
anthropologist . 

Revised  44  100  82  .32 

The  main  use  of  classification  of  races  is 

(a)  to  prove  the  existence  of  biological  races. 

(b)  to  aid  the  opponents  of  race  propaganda. 

(c)  to  discover  the  potentialities  of  each  race. 

-*(d)  to  serve  as  a statistical  concept  for  the 
anthropologist . 

Revision.  Distracters  have  been  made  perallel  in  form 
and  meaning. 
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Item  No.  25  N Pu  0 


Original  59  85  66  .23 

Barnes 

mainly 

envisions  that  prejudice  will  be  eliminated 
through 

*(a) 

education. 

(b) 

intermarriage . 

(c)  lecral  sanctions. 


(d) 

the  abolition  of  capitalism. 

Revised  44  84  82  .07 

Barnes 

mainly 

envisions  that  race  prejudice  will  he  eliminated 
through 

*(  a) 

education. 

(b) 

intermarriage . 

(c) 

legal  sanctions. 

(d) 

the  abolition  of  capitalism. 

Revision, 

Statement  has  been  clarified  by  use  of  "race". 
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Item  No.  26 


N Pu  Pi  0 


Original 


138  82  38  .44 


The  protocols  of  the  Elders  of  Zion  were  conclusively 
proved  to  he  forged  when 

(a)  the  author  made  a deathbed  confession  of  his  work. 

( b)  the  Tzarist  archives  were  examined  after  the 
Russian  Revolution. 

-*(c)  the  original  source  of  the  Protocols  was  discovered 
to  be  a French  satire. 

(d)  all  possible  investigations  of  the  validity  of  the 
documents  revealed  no  evidence  to  confirm  the 
statements  set  forth  in  the  Protocols. 


The  protocols  of  the  Elders  of  Zion  were  conclusively 
proved  to  be  forged  when 

( a)  the  author  made  a deathbed  confession  of  his  work. 

(b)  the  Tzarist  archives  were  examined  after  the 
Russian  Revolution. 

-*(c)  the  original  source  of  the  Protocols  was  discovered 
to  be  a French  satire. 

(d)  investigation  revealed  no  evidence  to  confirm  the 
validity  of  the  documents  or  their  contents. 


Revision . Distracters  have  been  made  uniform  in  length 
and  parallel  in  form  and  meaning. 


Revised 


44  87  25  .61 
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Item  No.  27  N Pu  Pi  0 

Original  39  62  26  .37 

Which  of  the  following  statements  best  describes  the 
attitude  of  the  AF  of  L toward  the  Negro? 

( a)  It  has  completely  ignored  them. 

(b)  It  has  actively  sought  to  organize  them. 

(c)  It  has  been  free  from  racial  discrimination. 

*(d)  It  has  been  prevailingly  hostile  to  the 
admission  of  Negroes. 

Revised  40  80  25  .54 

The  attitude  of  the  AF  of  L toward  the  Negro  has  been 
for  the  most  part  to 

(a)  ignore  them. 

(b)  organize  them. 

*(c)  exclude  them  from  membership. 

(d)  show  no  racial  discrimination. 


Revision 


Premise  has  been  rephrased  in  statement  form 
including  phrases  repeated  In  distracters. 
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Item  No.  28 


N Pu  Pi  0 


Original 


59  75  36  .39 


Which  of  the  following  reasons  was  not  enunciated 
by  the  Southerners  to  prove  the  goodness  of  slavery: 

(a)  The  Negroes  benefited  because  they  had  been 
rescued  from  savagery. 

(b)  The  plantation  owners  benefited  because  they  could 
devote  themselves  to  duties  of  government. 

(c)  American  society  as  a whole  benefited  because  there 
was  no  unrest. 

*(d)  Properly  interpreted,  slavery  did  not  violate  the 
spirit  of  the  Declaration  of  Independence* 


Which  of  the  following  reasons  was  not  enenciated  by  the 
Southerners  in  their  justification  of  slavery? 

(a)  The  Negroes  benefited  because  they  had  been 
rescued  from  savagery. 

(b)  The  plantation  owners  benefited  because  they  could 
devote  themselves  to  duties  of  government. 

(c)  American  society  as  a whole  benefited  because  there 
was  no  unrest. 

*(d)  The  U.S.  government  benefited  because  slavery  did 
not  violate  the  spirit  of  the  Declaration  of 
Independence . 


Revision.  1.  Statement  has  been  clarified. 


Revised 


44  100  61  .47 


2.  Distracters  have  been  made  parallel  in  form 
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Item 

No.  29 
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Pl  0 

Original 

59 

83 

59  .27 

To 

most  sociologists  migrations 

prove 

that 

racial  purity  is 

( a)  possible . 

*(  b)  a myth. 

(c)  a thing  of  bygone  days. 

(d)  limited  to  a few  African  tribes. 

Revised  44  91  61  .35 

To  most  sociologists  migrations  prove  that  racial  purity 
*(a)  never  has  existed. 

(b)  may  exist  in  the  future. 

(c)  existed  in  the  past. 

(d)  exists  in  certian  areas. 


Revision.  Distracters  have  been  made  uniform  in  length 
and  parallel  in  form  and  meaning. 
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Item  No.  30 


N Pu  Pi  0 


Origins! 


39  95  85  .19 


The  "Black  Codes"  refer  to  those  laws  passed  by 

(a)  Negro  churches  to  discipline  their  members. 

(b)  the  Par  West  to  govern  the  entrance  of  Negroes 
to  their  states. 

(c)  the  South  to  regulate  the  appalling  conditions 
of  the  Birmingham  coal  mines. 

*(d)  southern  legislatures  after  1865  to  control  the 
Negroes  who  had  been  freed. 


The  "Black  Codes"  refer  to  those  laws  passed  by 
the  State  legislatures  of 

(a)  the  North  before  1860  to  control  the  underground 
railroad . 

(b)  the  Par  West  after  1820  to  limit  immigration  of 
the  Negroes. 

(c)  the  South  after  1900  to  regulate  Negro  working 
conditions . 

*(d)  the  South  after  1865  to  control  the  emancipated 
Negroes . 


Revision.  Distracters  have  been  made  parallel  in  meaning. 


Revised 


40  95  80  .25 
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Item  No.  31 


N Pu  Pi  0 


Original 


59  93  31  .70 


According  to  Parkes,  the  main  explanation  of  why  many 
Jews  practiced  usury  in  the  Middle  Ages  is  that 

(a)  this  is  a trait  common  to  Jewry. 

*(b)  the  Christian  church  forbade  its  members  to 
nractice  it. 

(c)  they  hoped  to  cause  many  indebted  Christians  trouble. 

(d)  it  was  the  only  way  they  could  exercise  their 
desire  for  money  in  those  days. 


According  to  Parkes,  many  Jews  practiced  usury  in  the 
Middle  Ages  because 

(a)  they  desired  to  take  advantage  of  the  Christians. 

(b)  they  had  long  been  accustomed  to  practice  it. 

(c)  they  preferred  it  to  other  ways  in  which  they 
could  make  money. 

*(d)  they  were  not  forbidden  to  practice  it,  as  were 


Revision.  1.  Statement  has  been  simplified. 

2.  Distracters  have  been  made  parallel  in  meaninsr. 


Revised 


44  95  50  .49 
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Item  No.  32 


N Pu  Pi  0 


Original 


39  90  39  .51 


The  "grandfather  clause"  was  a method  of 

-"-(a)  disfranchising  the  Negro  in  the  South. 

(b)  restricting  the  social  activity  of  the  Negro. 

(c)  restricting  the  economic  opportunities  of  the  Negro. 

(d)  increasing  the  educational  facilities  of  the  Negro. 


The  "grandfather  clause"  was  a method  of  restricting  the 
Negro  in  the  South  in  respect  to 

(a)  social  life. 

•*(b)  political  activity. 

(c)  economic  opportunity. 

(d)  educational  advancement. 

Revision.  1.  Statement  has  been  rephrased  to  include 

phrases  from  distracters. 

2.  Distracters  have  been  made  uniform  in  length. 


Revised 


40  95  55  .46 
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Item  No.  33  N Pu  ?1  0 

Original  59  92  44  .50 

Which  of  the  following;  was  not  a contention  of  Gobineau? 

(a)  Each  race  has  a peculiar  characteristic. 

(b)  The  white  race  is  superior  to  the  other  two. 

(c)  There  are  three  primary  races  - white,  black,  and 
yellow . 

*(d)  The  term  Aryan  should  be  confined  to  linguistic 
reference . 

Revised  44  66  55  .17 

Which  of  the  following  was  not  a contention  of  Gobineau? 
*(a)  The  white  race  is  in  decline. 

(b)  Each  race  has  a peculiar  characteristic. 

(c)  The  white  race  is  superior  to  the  other  two  races. 

(d)  There  are  three  primary  races  - white,  black,  and 
yellow. 


Revision.  Distracters  have  been  made  parallel  in  meaning. 
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Item  No . 34  N Pu  Pi  0 

Original  40  98  55  .50 

Which  of  the  following  reasons  best  accounts  for  the 
Negro  migration  to  the  North  after  1914? 

*( a)  Demand  for  Negro  labor  in  the  North. 

(b)  The  increase  in  Negro  prejudice  during  the  war  years. 

(c)  The  campaign  of  labor  unions  urging  Negroes  to 
migrate'  northward. 

(d)  Du  Bois  and  other  Negro  leaders  urged  Negroes  to 
migrate  to  the  North. 

Revised  39  90  62  .32 

Negro  migration  to  the  North  after  1914  was  due  chiefly  to 

*(  a)  the  demand  for  negro  labor  in  the  North. 

(b)  the  increase  in  negro  prejudice  in  the  South. 

(c)  the  request  of  labor  unions. 

(d)  the  advice  of  negro  leaders. 

Revision . 1.  Premise  has  been  rephrased  in  statement  form 

to  include  phrases  from  the  distracters. 

2.  Distracters  have  been  made  parallel  in  form 
and  meaning. 
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Item  No.  35 


N Pu  Pi  0 


Original 


40  100  60  .47 


The  concept  of  the  "marginal  man"  refers  to  that 
individual  who 

(a)  has  no  distinctive  nationality. 

(b)  is  not  a citizen  of  any  country. 

*(c)  is  without  a "race"  or  a society. 

(d)  lives  on  the  margin  of  subsistence. 


The  concept  of  the  "marginal  man"  refers  to  that 
individual  who  is  without 

■*(  a)  "race"  or  society. 

(b)  distinctive  nationality. 

(c)  adequate  subsistence. 

(d)  citizenship  in  any  country. 

Revision.  1.  Statement  has  been  rephrased  to  include 


phrases  from  distracters. 

2.  Distracters  have  been  made  uniform  in  length 
and  parallel  in  form  and  meaning. 


Revised 


39  93  56  .43 
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Item  No.  36 
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Original 
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.38 

The  great  depression  of  1929 

(a)  affected  the  Negro  less  than  other  Americans. 

-*(b)  constituted  a crisis  in  the  life  of  the 
American  Negro. 

(c)  administered  a severe  blow  to  white  collar  Negroes 
but  not  to  those  engaged  in  agriculture. 

(d)  did  little  harm  to  the  occupation  in  which  the 
bulk  of  the  Negroes  were  employed. 

Revised  39  77  49  .28 

During  the  great  depression  of  1929,  the  American 
Negro  encountered 

(a)  less  difficulty  than  other  Americans. 

*(b)  a critical  period  in  his  life. 

(c)  more  trouble  in  white  collar  work  than  in 
agriculture . 

(d)  little  occupational  difficulty. 

Revision.  1.  Statement  has  been  rephrased  to  include 

phrases  from  distracters. 

2.  Distracters  have  been  made  uniform  in 
length  and  parallel  in  meaning. 
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Item  No.  37 


Socially, 

( a)  as 

( b)  as 

*(c)  more  favorably  than  a dark-skinned  negro 
(d)  less  favorably  than  a dark-skinned  negro 
Revised  40  93  85 

Socially,  the  whites  look  upon  the  mulatto  as 
(a)  equal  to  the  dark-skinned  negro. 

Mb)  superior  to  the  dark-skinned  negro. 

(c)  inferior  to  the  dark-skinned  negro. 

(d)  None  of  the  above. 


N Pu  Pi 

Original  39  97  85 

the  whites  look  upon  the  mulatto 
eaual  to  themselves, 
equal  to  the  dark-skinned  negro. 
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.21 


.15 


Revision . 


Distracters  have  been  made  uniform  in  length 
and  parallel  in  meaning. 
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Item  No.  38 


N Pu  Pi  0 


Original 


138  29  17  .15 


According  to  Young,  the  elimination  of  prejudice  is 
(a)  at  the  present  time  proceeding  very  well. 

■*(b)  only  an  ideal  for  which  we  should  strive. 

(c)  attainable  mainly  through  the  educational  system. 

(d)  fairly  simple  since  very  little  real  prejudice 


According  to  Young,  the  elimination  of  prejudice  is 
(a)  a problem  which  is  being  solved  in  the  present. 

•M-(b)  an  ideal  which  should  be  striven  for  in  the  future. 

(c)  a question  for  continuous  educational  treatment. 

(d)  a matter  of  no  immediate  concern. 


Revision.  Distracters  have  been  made  parallel  in  meaning. 
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Item  No.  39 


Original 


N Pu  Pi  i> 

103  69  21  .48 


The  classification  of  races  has  been  set  up  as  a result 
of  study  ins; 

(a)  complete  genealogies  in  the  archives. 

(b)  the  fossils  uncovered  by  archaeologists. 

*(  c)  existing;  differences  in  present  day  adults. 

(d)  the  genetic  traits  in  the  germ  plasm  of 
various  individuals. 

Revised 

No  change. 


Item  No.  40 
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Original 

103 

82 
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The  sterotype  depicted  in  Thomas 
The  Klansmen,  is  that  of  a 

Nixon 

»s  : 

novel. 

(a)  happy  negro. 

(b)  comic  negro. 
*(c)  bestial  negro. 

(d)  tragic  negro. 

Revised 


No  change 
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Item  No.  41 


N P 
79  100 


P 

69 


0 

.42 


Which  of  the  following  belongs  to  the  Mongoloid  stock? 

( a)  Alpine . 

(b)  Bushman. 

(c)  Armenoid. 

*(d)  Polynesian. 

Revised 


No  change. 


Item  No.  42 
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Original 

103 

93 

58 

In  the  scale  of  nationality  preferences  taken 
among  U.S.  college  students,  the  Turks  appeared 

( a)  not  at  all . 

(b)  near  the  top. 

*(c)  near  the  bottom. 

(d)  in  the  middle. 

Revised 

✓ 

No  change. 
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Item  No.  45  N Pu  Pi  0 

Original  103  96  65  .38 

It  is  believed  that  the  Rust  mechanical  cotton  picker  will 

(a)  improve  the  status  of  Negro  agricultural  labor. 

(b)  improve  the  agricultural  production  of  New  England. 

-*(c)  harm  the  position  of  Negro  agriculture  in  the  South. 

(d)  harm  the  cotton  bolls  more  than  the  present 
system  of  picking. 

Revised 

No  change. 


Item  No.  44 
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Original 
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.33 

Prejudices 

develop  in  most 

instances 

as  a result 

of 

( a)  shock. 

•M-(b)  imitation. 

(c)  experience. 

(d)  innateness. 

Revised 

No  change. 
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Item  No.  45 


Original  103  100  80  .32 

The  "100  per  cent  Americanism"  slogan  that  sprung  up 
during  World  War  I refers  to  the  fact  that  the 
United  States  had  a prejudice  against 

( a)  Jews . 

(b)  Catholics. 

(c)  Japanese. 

*(d)  Foreigners. 

Revised 

No  change. 


Item  No.  46  N Pu  P]_  0 

Original  103  61  28  .32 

According  to  Barnes,  our  largest  minority  group  today 
outside  of  the  Negro  is  the 

( a)  Indian . 

(b)  Chinese. 

*(c)  Mexican. 

(d)  Japanese. 

Revised 


No  change 
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Item  No.  47 

Original 

The  hair  and  lip  formation 
that  they  are 

(a)  less  primitive  than 

(h)  less  primitive  than 

(c)  more  primitive  than 

*-(d)  None  of  the  above. 

Revised 

No  change. 


N P P 0 
103  58  28  .30 

of  the  Mongoloid  stock  proves 

the  Negroid  stock, 
the  Caucasian  stock, 
the  Caucasian  stock. 


Item  No. 

Parkes 

(a) 

(b) 

*(c) 

(d) 


48  N P P 0 

Original  103  69  45  .25 

points  out  that  the  bulk  of  the  German  people 

saw  through  Hitler's  antisemitic  scheme. 

accepted  antisemitic  propoganda  because  they  feared 
the  secret  police. 

believed  that  the  Jews  were  a vast  and  hostile 
world  force. 

believed  that  it  was  a cruel  but  necessary  step  to 
eliminate  the  Jews. 

Revised 


No  change 
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Item  No.  49  N Pu  P]_  0 

Original  79  88  70  .24 

In  the  year  1932  the  unity  of  race  conscious  groups 


*(  a) 

increased. 

(b) 

was  slight. 

(c) 

disappeared. 

(d) 

remained  unchanged. 
Revised 

No  change. 

Item  No.  50  N Pu  P]_  $ 

Original  142  77  57  .23 

Barnes  points  out  that  the  Americans  show  the  most 
prejudice  towards  the 


( a) 

Indian. 

(b) 

Chinese . 

(c) 

Mexican . 

*(d) 

J apanese . 

Revised 

No  change 
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Item  No.  51  N Pu  P]_  $ 

Original  105  97  86  .22 

The  main  occupation  of  the  Negro  up  to  1914  was  in 

(a)  industry  and  mining. 

*(b)  agriculture  and  domestic  service. 

(c)  the  rising  factories  of  the  South. 

(d)  domestic  service  and  transportation. 

Revised 

No  change. 


Item  No.  52 

N Pu 

Pi 

0 

Original 

123  52 

33 

.19 

Physical  differences  between 

stocks  prove 

that 

races  are 

( a)  mixed . 

( b)  unequal. 

( c )  different . 

(d)  pure. 

Revised 

No  change. 
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Item  No.  53 


Original 


N Pu  Pi  0 

103  71  59  .13 


On  the  basis  of  present  evidence  it  is  believed  by  most 
sociologists  that  races  are 

(a)  unequal. 

(b)  definitely  equal. 

■fr(c)  apparently  equal. 

(d)  None  of  the  above. 

Revised 

No  change • 


Item  No.  54 


Original 


N 


u 


pi  a 


103  53  43 


.10 


One  of  the  most  common  devices  among  upper  class  Negroes 
for  escaping  caste  implications  is  the  development  of 

( a)  servility. 

(b)  repression. 

(c)  aggression. 

•*(d)  complacency. 

Revised 

No  change . 
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Item  No,  55 


Original 


N 

103 


Pu 

11 


Pi  0 
20  .14 


People  that  believe  in  racial  superiority  belong  to  the 

(a)  Crobineauis  ts  . 

(b)  race  purists. 

(c)  race  propagandists. 

*(d)  race  conscious  groups. 

Revised 

No  change. 
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APPENDIX  D 
TEST  NO. 2 

TABULATED  DATA 

ORIGINAL  AND  REVISED  ITEMS 
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Table  8. 

Phi  Coefficients  of  Validity 
Original  and  Revised  Items 
Test  No.  2 


Original 

Form 

.Revised 

Form 

Item 

% 

i 

phi 

% 

% 

phi 

No. 

upper 

lower 

upper 

lower 

group 

group 

group 

group 

passing 

passing 

passing 

passing 

(N  35) 

(N  35) 

(N  35) 

(N  35) 

1 

100 

92 

.20 

100 

100 

.00 

2 

100 

97 

.35 

97 

71 

.42 

3 

26  - 

20 

.07 

6 

14 

- .13 

4* 

66 

20 

.47 

63 

11 

.54 

5 

46 

9 

.41 

37 

26 

.12 

6 

51 

37 

.14 

63 

6 

.60 

7 

57 

23 

.35 

54 

17 

.39 

8* 

100 

83 

.30 

100 

83 

.30 

9 

49 

43 

.06 

63 

26 

.37 

10 

94 

51 

.43 

94 

43 

.55 

11 

41 

34 

.07 

43 

29 

.15 

12 

51 

51 

.00 

71 

66 

.05 

13 

83 

40 

.47 

86 

26 

.61 

14 

74 

14 

.34 

60 

26 

.34 

15* 

94 

86 

.08 

94 

33 

.16 

16* 

89 

31 

.59 

80 

29 

.51 

17 

83 

51 

.34 

57 

31 

.26 

18* 

97 

80 

.27 

94 

66 

.35 

19 

94 

80 

.21 

97 

92 

.09 

20 

34 

31 

.03 

29 

17 

.14 

21 

97 

57 

.43 

92 

46 

.50 

22* 

89 

74 

.19 

89 

80 

.11 

23 

60 

11 

.51 

60 

9 

.54 

24 

86 

26 

.66 

89 

57 

.36 

25 

72 

29 

.43 

86 

34 

.53 

26 

26 

14 

.15 

29 

11 

.22 

27 

69 

26 

.43 

74 

20 

.54 

28 

51 

11 

.43 

54 

20 

.35 

29 

43 

23 

.21 

31 

06 

.32 

30 

80 

51 

.31 

94 

43 

.55 
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Table  8.  (concluded) 


Original 

Form 

Revised 

Form 

Item 

No. 

% 

upper 
ffroup 
passing 
(N  35) 

% 

lower 
sroup 
passins 
(N  35) 

phi 

% 

upper 
s:roup 
oassing 
(N  35) ' 

% 

lower 
group 
passing 
(N  35) 

£hi 

31 

92 

40 

.45 

94 

46 

.52 

32 

97 

29 

.71 

86 

26 

.61 

33 

92 

46 

.50 

100 

57 

.53 

34 

97 

89 

.16 

100 

83 

.32 

35 

29 

14 

.18 

23 

29 

- .07 

36  * 

54 

20 

.35 

71 

26 

.45 

37 

37 

3 

.42 

20 

3 

.29 

38* 

40 

9 

.36 

31 

3 

.37 

39 

86 

34 

.52 

71 

11 

.66 

40 

40 

14 

.29 

57 

6 

.55 

41 

60 

11 

.59 

66 

29 

.37 

42 

60 

29 

.31 

92 

89 

.05 

43 

63 

20 

.44 

69 

23 

.43 

44* 

54 

14 

.41 

71 

14 

.58 

45 

51 

6 

.49 

77 

9 

.68 

46 

54 

29 

.25 

49 

11 

.41 

47 

69 

34 

.35 

49 

14 

.38 

48 

69 

20 

.49 

69 

14 

.56 

49 

69 

17 

.53 

66 

23 

.43 

50 

92 

43 

.52 

80 

40 

.41 

51 

74 

34 

.40 

83 

23 

.60 

52 

77 

11 

.67 

69 

20 

.49 

53 

31 

20 

.13 

23 

23 

.00 

54* 

89 

74 

.19 

86 

80 

.08 

55 

57 

63 

- .06 

51 

51 

.00 

56* 

77 

49 

.29 

69 

49 

.20 

57 

46 

6 

.45 

57 

20 

.38 

58 

23 

3 

.30 

34 

20 

.16 

59 

37 

26 

.12 

37 

26 

.12 

60 

40 

9 

.36 

14 

23 
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Table  9. 

Distribution  of  Phi  Coefficients  of  Validity 
of  Test  Items  in  Test  No.  2 


Ph 

1 \ Original 

Revised 

Form 

Unchanged 

\ 

Form 

Horizontal 

Response 

Vertical 

Response 

.71 

.75 

1 

0 

0 

0 

.66 

- 

.70 

1 

1 

0 

0 

.61 

- 

.65 

0 

3 

0 

0 

.56 

- 

.60 

1 

3 

1 

1 

.51 

- 

.55 

4 

7 

0 

2 

.46 

- 

.50 

4 

2 

1 

0 

.41 

- 

.45 

9 

5 

1 

1 

.36 

- 

.40 

2 

6 

1 

1 

.31 

- 

.35 

7 

4 

1 

1 

.26 

- 

.30 

3 

3 

3 

1 

.21 

- 

.25 

3 

1 

0 

0 

.16 

- 

.20 

3 

1 

2 

2 

.11 

- 

.15 

4 

4 

0 

1 

.06 

- 

.10 

3 

1 

1 

1 

.01 

- 

.05 

1 

2 

0 

0 

-.04 

- 

.00 

1 

3 

0 

0 

- .09 

- 

- .05 

1 

1 

0 

0 

-.14 

- 

-.10 

0 

2 

0 

0 

N 

4Q 

49 

11 

13 - 

Me8n 

S.D. 

.328 

.179 

.332 

.208 

.303 

.144 

.330 

.173 
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9 VI. 

Item 

No. 

1 

N 

Pu 

Pi 

0 

Original 

35 

100 

92 

.20 

If 

you 

break  a school  window, 

you  should- 

*1  tell  the  principal  you  did  it  2 repair  it 
3 say  nothing  4 say  you  didn't  do  it 
5 say  somebody  else  did  it 

Revised  35  100  100  .00 

If  you  break  a school  window,  you  should  say  that 

1.  you  did  it. 

2.  you  didn't  do  it. 

3.  somebody  else  did  it. 

4.  you  know  nothing  about  it. 

5.  vou  are  not  concerned. 


Revi s ion 


Distracters  have  been  made  parallel  in  meaning 


i i 
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Item 

No. 

2 

N Pu  Pi 

0 

Original 

35  100  77 

.35 

If 

your 

political  party  nominates 

a man  you  know 

is  unfit 

for  the  office,  you  should- 

1 vote  for  him  anyway  2 resign  from  your  party 
3 not  vote  at  all  *4  vote  for  the  candidate  who  you 
think  is  the  best  one  5 vote  for  the  candidate 
likely  to  win 

Revised  35  97  71  .42 

If  your  political  party  nominates  a man  you  know  is  unfit 
for  the  office,  you  should  vote  for 

1.  your  party’s  candidate. 

2.  no  candidate  at  all. 

3.  the  candidate  most  likely  to  win. 

4.  the  best  looking  candidate. 

*5.  the  best  qualified  candidate. 

Revision . 1.  Premise  has  been  reworded  to  include  phrases 

from  the  distracters. 


2.  Distracters  and  correct  answer  have  been 
made  parallel  in  form  and  meaning. 
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Item  No.  3 

N 

Pu 

Pi 

0 

Original 

35 

26 

20 

.07 

The  Puri tens - 


1 were  the  Separatists  2 settled  in  Holland 
3 were  opposed  to  war  *4  settled  the  Colony  of 
Massachusetts  5 settled  the  Plymouth  Colony 


Revised  35  6 14  -.13 

In  the  early  1600* s,  there  were  many  English  people 
who  wished  to  purify  the  Church  of  England.  Prom 
these  came 

1.  the  Separatists. 

2.  emigrants  to  Holland. 


3.  the  pacifists. 

*4.  the  settlers  of  Massachusetts  Bay  Colony. 
5.  the  settlers  of  the  Plymouth  Colony. 


Revision.  l.Item  has  been  made  a more  direct  measure  of 
attainment  by  substituting  a description  in 
place  of  the  term  "Puritan". 

2 .Dis trac ters  and  correct  answer  have  been  mad 
paralled  in  form  and  meaning. 
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Item  No.  4 


Original 


N 

35 


Pu 

66 


Pi 

20 


0 

.47 


The  first  fierhting  of  the  Revolutionary  War  took  place  in- 

1 New  York  2 Virginia  3 Connecticut 
*4  Massachusetts  5 South  Carolina 


Revised  35  63  11  .54 

The  first  fighting*  of  the  Revolutionary  War  took  place  in 

1 . New  York. 

2.  Virginia. 

3.  Connecticut. 

•*4 . Massachusetts. 

5.  South  Carolina. 


Revision.  No  chancre. 
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Item  No.  5 N Pu  P]_  0 

Original  35  46  9 .41 

A state  that  was  not  one  of  the  Thirteen  Colonies  is- 

1 New  Jersey  -*2  Maine  3 Delaware  4 Georgia 
5 Pennsylvania 

Revised  35  37  26  .12 

One  of  the  Thirteen  Colonies  was 

1.  Maine. 

2.  West  Virginia 

3.  Florida. 

*4 . Georgia. 

5.  Kentucky. 


Revision 


Item  has  been  made  positive  instead  of  negative 
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Item  No.  6 N Pu  P]_  0 

Original  35  51  37  .14 

As  a result  of  Cabot's  explorations- 

1 he  founded  colonies  in  America  *2  England  claimed 
North  America  3 England  went  to  war  with  Spain 
4 the  Line  of  Demarcation  was  determined  5 he  made 
peace  wi th  the  Indians 


Revised  35  63  6 .60 

John  Cabot,  as  a result  of  his  explorations, 

1.  founded  colonies  in  America. 

*2 . claimed  North  America  for  England. 

3.  forced  England  into  war  with  Spain. 

4.  determined  the  Line  of  Demarcation. 

5.  made  peace  with  the  Indians. 


Revision . 1.  Premise  has  been  reworded  to  include  phrases 

from  the  distracters. 


2.  Distracters  have  been  made  parallel  in  form 
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Item  No.  7 
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Original 

35 

57 

23 

.35 

When  Henry  Hudson  discovered 
the  flag  of- 

New  York, 

he  was  i 

sailing  under 

1 Italy  2 England 

*3 

Holland 

4 Spain 

5 Portugal 

Revised 

35 

54 

17 

.39 

The  discoverer  who  first  explored  the  Hudson  River  was 
sailing  under  the  flag  of 

1.  Italy. 

2.  England. 

*3.  Hollend. 

4.  Spain. 

5.  Portugal. 


Revision . Item  has  been  made  a more  direct  measure  of 
attainment  by  substituting  the  "deed  for  the 
doer" . 
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Item  No,  8 
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Original  35  100  83 

The  telephone  was  invented  by- 
*1  Bell  2 Morse  3 Field  4 Edison  5 

Revised  35  100  83 

The  telephone  was  invented  by 
*1,  Bell. 

2.  Morse. 

3.  Field. 

4.  Edison. 

5.  Marconi. 


.30 

Marconi 


.30 


Revision 


No  change 
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Item  No.  9 
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Original 
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35 

49 
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An  important  event  that  hastened  the  Revolutionary  War  was- 

1 the  capture  of  New  York  by  the  British 

2 Bacon's  Rebellion  3 the  repeal  of  the  Stamp  Act 
-*4  the  Boston  Tea  Party  5 the  Whiskey  Rebellion 


Revised  35  63  26  .37 

An  imoortant  event  which  led  up  to  the  outbreak  of  the 
Revolutionary  War  was 

1.  the  capture  of  New  York. 

2.  the  repeal  of  the  Stamp  Act. 

3.  Bacon's  Rebellion. 

*4.  the  Boston  Tea  Party. 

5.  the  Wills  key  Rebellion. 


Revision 


The  statement  in  the  premise  has  been  clarified 
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Item  No.  10 
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The  "Forty-niners"  were- 

1 settlers  in  Texas  2 men  who  fought  for  Oregon 
3 a regiment  in  the  Revolutionary  War  -w-4  seekers  for 
gold  in  California  5 a regiment  in  the  Civil  War 


Revised  35  94  43  .55 

The  term  "Forty-niners"  refers  to 

1.  settlers  in  Texas. 

2.  men  who  fought  for  Oregon. 

3.  a Revolutionary  War  regiment. 

•*4 . gold  seekers  in  California. 

5.  a Civil  War  regiment. 

Revision . 1.  The  statement  in  the  premise  has  been  made 

more  definite. 

2.  Distracters  have  been  made  parallel  in  form. 
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Item  No.  11 
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Original 

35 

41 

34 

< 

.07 

The  French  had  trouble  in  settling  America  chiefly  because  cf- 

1 lack  of  money  2 the  severe  climate  *-3  the  enmity  of 
the  Iroquois  4 lack  of  food  5 lack  of  leaders 

Revised  35  43  29  .15 

One  of  the  chief  obstacles  which  the  French  experienced  in 
their  attempts  to  settle  America  was 

1 . lack  of  monev . 

2.  the  severe  climate. 

-*3.  enmity  of  the  Iroquois. 

4.  lack  of  food. 

5.  lack  of  leaders. 


Revision 


The  statement  in  the  premise  has  been  more 
precisely  stated. 
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Item  No.  12 


N 


0 


Fu  ?1 

Original  55  51  51  .00 

Slaves  were  first  introduced  into  the  colonies- 

1 because  cheap  labor  was  needed  to  grow  tobacco 

2 to  grow  corn  *3  to  grow  cotton  4 to  fight  the 

Indians  5 to  build  roads 

Revised  35  71  66  .05 

Slaves  were  first  introduced  into  the  colonies 

1.  to  grow  corn. 

2.  to  grow  tobacco. 

*3.  to  grow  cotton. 

4.  to  fight  the  Indians. 

5.  to  build  roads. 


Revision 


The  distracters  have  been  made  parallel  in  form 
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Item  No.  13 
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Pu  Pi  0 

Original  35  83  40  .47 

Several  important  wars  in  the  American  colonies  were  fought 
between-  1 France  and  Spain  -*2  France  and  England 

3 England  and  Holland  4 England  and  Spain 
5 Holland  and  Spain 


Revised  35  86  26  .61 

The  campaign  which  ended  in  Braddock's  defeat  occurred  in  a 
war  between 

1.  France  and  Spain. 

#2.  France  and  England. 

3.  England  and  Spain. 

4.  England  and  Holland. 

5.  Holland  and  Spain. 

Revision . The  item  has  been  made  a more  direct  measure  of 
attainment  by  substituting  a specific  event  for 
the  general  term  "wars". 
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Item  No.  14 
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While  he  was  President,  Washino-ton  succeeded  in  having- 

1 war  with  England  2 the  Erie  Canal  finished 
*3  peace  with  all  countries  4 railroads  built 

5 war  with  Spain 

Revised  35  60  26  .34 

Duriner  Washington’s  term  as  President,  the  United  States 
was  at  war  with 

1.  England. 

2.  Spain. 

3.  Prance. 

*4.  no  country  at  all. 

5.  Mexico. 


Revision . 1.  The  statement  in  the  premise  has  been  reworded 

to  include  phrases  from  the  distracters. 

2.  The  distracters  have  been  made  parallel  in 
form  and  meaning. 


‘>J  . o r ^©  -*1 

• ’ r - ',.«!«•  t 1 L ' Vi 
i 3I1  r*.  "If  is”/  I 


f;?  ' <r  *i£vr  e 3W 
£-  - . £ 

. rrl  o . 2 

. a -rr  ■ r . 


. 

. ' ' © . 7 


in©iE setose  ariT  .1 
► »ffP  • S 

. < iVi?©-  / ! ■ *rr?c ■' 


Item  No.  15  N Pu  ^1  0 

Original  35  94  86  .08 

Community  health  depends  mo3t  directly  on- 

1 a high  tax  rate  *2  a pure  water  supply 
3 good  railroad  transportation  4 good  schools 
5 a good  police  force 

Revised  35  94  83  .16 

Community  health  depends  most  directly  on 
1.  a high  tax  rate. 

*2.  a pure  water  supply. 

3.  a good  transportation  system. 

4.  a good  school  svstem. 

5.  a good  police  force. 


Revision 


No  change 
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Item  No.  16 
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An  important  city  obtained  with  the  Louisiana  Purchase  was- 

1 Cincinnati  2 Pittsburgh  3 Louisville 
*4  New  Orleans  5 Terre  Haute 


Revised  35  30  29  .51 

An  important  city  obtained  with  the  Louisiana  Purchase  was 

1.  Cincinnati. 

2.  Pittsburgh. 

3.  Louisville. 

-*4 . New  Orleans. 

5.  Terre  Haute. 


Revision.  No  change 
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Item  No.  17 


Original 


N 

55 


As  a result  of  the  French  and  Indian  War- 

1 England  lost  much  land  2 the 
independent  *3  France  lost  r 
America  4 the  Indians  were  driven  from  the  East 
5 Mexico  became  independent 


Pu 
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33 

51 

.34 

War- 

olonists 

became 

it  of 

her 

possessions  in 

Revised  35  57  31  .26 

The  French  and  Indian  War  resulted  in  loss  of  territory  for 
1.  England. 

*2.  France. 

3.  Spain. 

4.  the  Indians. 

5.  Mexico. 

Revision.  1. Premise  has  been  reworded  to  include  phrases 

from  the  distrscters. 

2.  Distracters  have  been  made  parallel  in  form 
and  meaning. 
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Item  No.  18 


Original 


Money  is  coined  by  the- 
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35 
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.27 

1 Department  of  Commerce  *2  Treasury  Department 
3 War  Department  4 Department  of  Justic 

5 Department  of  Taxation 


Revised  35  94  66  .35 

Money  is  coined  by  the 

1.  Department  of  Taxation. 

*2.  Treasury  Department. 

3.  War  Department. 

4.  Department  of  Commerce. 

5.  Department  of  Justice. 


Revision 


No  chance 
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Item  No.  19 
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35 
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.21 

To  become  an  American  citizen,  an  immigrant  must- 

1 go  to  school  2 save  money  *3  be  naturalized 
4 own  land  5 become  popular 


Revised  35  97  92  .09 

A foreien-born  person  who  wishes  to  become  an  American 
citizen  must 

1.  be  educated. 

2.  be  wealthy. 

*3.  be  naturalized. 

4.  own  land. 

5.  be  popular. 


Revision 


Item  has  been  made  a more  direct  measure  of 
attainment  by  substituting  for  the  term 
"immi errant " its  definition. 


. 


. ’ 

. aaol«iercA  as  ©mooed  oT 

. 

5^  X cvx’(  > 


. » T * ? 1 « 

. ~ r'r * f od  . 3 

« jb©s  -la an  ©d  . 


. '3  o f *-  d . 


sen  need  e«n  raedX 

♦ 


164 


Item  No.  20 


N Pu  Pi  t 


Original 


35  34  31  .03 


The  first  railroad  to  .join  the  East  and  the  West  was 
f inished- 

1 about  1350  2 about  1390  3 after  1900 


The  first  railroad  to  .join  the  East  and  the  West  was 
finished 

1.  about  1350. 

2.  about  1860. 

*3 . about  1370. 

4.  about  1890. 

5.  after  1900. 

Revision.  Distracters  have  been  made  parallel  in  form. 


*4  just  after  the  Civil  War 


5 about  1360 


Revised 


35  29  17  .14 


so. 


. : . 

..  C 

. 

. • ' furde  . 

.0091  .6 


. - 


Item  No.  21 

N 

?u 

Pi 

Original 

35 

34 

31 

The  most 

important  Southern 

general  in 

the 

Civil 

1 Beauregard.  2 Early  *3  Lee  4 Meade 


Revised  35  29  17 

The  most  important  Confederate  general  was 

1.  Beauregard. 

2.  Early. 

*3.  Lee. 

4.  Meade. 

5.  Jackson. 


0 

03 

War  was- 
5 Grant 

14 


Revision 


The  statement  in  the  oremise  has  been  clarified 
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Item  No.  22  N Pu  0 

Original  55  89  74  .19 

The  handling  of  the  mails  is  controlled  by  the- 

1 Treasury  Department  2 Department  of  the  Interior 
3 Department  of  Commerce  *4  Post  Office  Department 

5 Department  of  Railroads 

Revised  35  89  80  .11 

The  handling  of  the  mails  is  controlled  by  the 

1.  Treasury  Department. 

2.  Department  of  the  Interior. 

3.  Department  of  Commerce. 

*4.  Post  Office  Department. 

5.  Department  of  Railroads. 

Revision.  No  change. 
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Item  No.  23 


N Pu  Pi  0 


Original  35  60  11  .51 

Right  after  the  Civil  War  the  South  was  controlled  by  the- 

1 Negroes  2 Confederate  Army  #3  Northern  politicians 
4 Ku  Klux  Elan  5 Democrats 


Revised  35  60  9 .54 

During  the  period  of  Reconstruction  the  South  was 
controlled  by  the 

1.  Negroes. 

2.  Confederate  Army. 

*3.  Northern  politicians. 

4.  Ku  Klux  Elan. 

5.  Democrats. 


Revision.  Item  has  been  made  a more  direct  measure  of 

attainment  by  using  the  term  "Reconstruction" 
instead  of  the  more  easily  identified  period 
"after  the  Civil  War". 
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Item  No.  24 


N Pu  Pi  0 


Original 


35  86  66  .26 


If  most  citizens  do  not  like  a law,  they  should- 

1 refuse  to  pay  their  taxes  *2  try  to  have  it  repealed 
3 not  obey  it  4 try  to  get  others  to  disobey  it 

5 move  to  another  city 


If  the  people  of  the  United  States  do  not  believe  in  the 
Federal  Income  Tax  law,  they  should 

1.  refuse  to  pay  the  tax. 

*2.  try  to  have  the  law  repealed. 

3.  advise  others  not  to  pay. 

4.  move  to  another  country. 

5.  pay  part  of  the  tax. 

Revision . Item  has  been  made  a direct  measure  of  attainment 
by  calling  for  the  application  of  knowledge. 


Revised 


35  89  57  .36 
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Item  No.  25 
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Original  55  72  29  .43 

The  community  that  depends  most  on  other  people  for  its 
food  and  clothing  is  the- 


*1  large  city  2 farming  village  3 frontier  settlement 
4 primitive  village  5 country  town 


Revised  35  86  34  .53 

The  community  that  depends  most  on  other  people  for  the 
raw  materials  for  its  food  and  clothing  is  the 

*1.  large  city. 

2.  farming  village. 

6,  frontier  settlement. 

4.  primitive  village. 

5.  country  town. 

Revision . The  statement  in  the  premise  has  been  made  more 
precise . 
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Item  No.  26 
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Original  35 

The  French  decided  to  help  the  colonies  after  the  battle  of- 

1 Brandywine  2 Trenton  3 Yorktown  *4  Saratoga 

5 Quebec 


Revised  35  29  11  .22 

The  success  of  Benjamin  Franklin’s  mission  to  France 
during  the  Revolutionary  War  followed  the  battle  of 

1.  Brandywine. 

2.  Trenton. 

3.  Yorktown. 

*4.  Saratoga. 

5.  Quebec. 


Revision 


Item  has  been  made  a more  direct  measure  of 
attainment  by  using  the  specific  instance 
rather  than  the  general  statement. 


H ' .oH 

2S. 

. -n  ' -nfhi.eT  . I 
• nc-.tn*  " . £ 

. fT  7C  '*>  'O’  . 

. pc  ri  . 

. 0*cfftu£  . r, 

. 


Item  No. 

27 

N 

Pu 

Pi 

0 

Original 

35 

69 

26 

.43 

Before 

Texas 

became  part  of 

the  United 

States- 

1 only  Americans  settled  it  2 there  were  no  Americans 
in  it  *3  many  Americans  settled  there  4 Americans 
bought  all  of  Texas  5 Americans  were  opposed  to 
settling  Texas 

Revised  35  74  20  .54 

Before  Texas  became  part  of  the  United  States,  Americans 
were 

1.  its  only  settlers. 

2.  none  of  its  settlers. 

*3.  numerous  among  its  settlers. 

4.  its  only  ovmers . 

5.  opposed  to  its  settlers. 


Revision . 1. Premise  has  been  reworded  to  include  a phrase 
from  the  distracters. 


2.  Distracters  have  been  made  parallel  in  form. 
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Item  No.  28 
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Original 

35 

51 

11 

.43 

The  two  states  of  continental 

United 

States  last 

admitted 

to  the  Union  are  in  the- 

1 Northwest  2 South  3 Southeast  *4  Southwest 

5 Northeast 

Revised  35  54  20  .35 

The  47th  and  48th  states  admitted  to  the  Union  are  located 
in  the 

1.  Northwest. 

2 . South. 

■*3.  Southwest. 

4.  Southeast. 

5.  Northeast. 


Revision 


Item  has  been  made  a more  direct  measure  of 
attainment  by  referring  to  the  states  by  number 
instead  of  identifying  them  as  the  last. 


nte  1*  0 


5 . 

. ■ ' mf-  ' i 


?■  i o.  r!  ri  • c 


. 

. .1  r?  :*wj  ' . f 

. ll  C l.  . 


. I-V  . > 

■ 

. t : -r  c . 

. 


. 


Item  No.  29 
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When  the  Revolution  started,  most  of  the  colonists  wanted- 

1 independence  of  Great  Britain  **2  Great  Britain  to 
respect  their  rights  3 Great  Britain  to  pay  back  all 
their  taxes  4 to  form  the  United  States  5 education 
for  all 

Revised  35  31  6 .32 

When  the  Revolution  started,  most  of  the  colonists  wished 
Great  Britain  to 

1.  give  them  their  independence. 

*2.  respect  their  rights. 

3.  pay  back  their  taxes. 

4.  form  the  United  States. 

5.  provide  education  for  all. 

Revision . 1.  Premise  has  been  reworded  to  include  a phrase 

from  the  distracters. 


2.  Distracters  have  been  made  parallel  in  form 
and  meaning. 
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Item  No.  30 
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.31 


Money  should  not  he  given  to  beggars  because- 

*1  it  will  make  them  keep  on  begging  2 more  people  will  beg 
3 they  will  hoard  it  4 they  will  use  it  improperly 

5 they  do  not  need  it 

Revised  35  94  43  .55 

Money  £iven  to  beggars  encourages  them 
*1.  to  keep  on  begging. 

2.  to  beg  in  greater  numbers. 

3.  to  hoard  it. 

4.  to  use  it  improperly. 

5.  to  t8ke  what  they  do  not  need. 

Revision.  1.  Premise  hss  been  made  positive. 

2.  Distracters  have  been  made  parallel  in  form 
and  meaning. 
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Item  No.  31  N Pu  P]_  0 

Original  35  92  40  .55 

Much  money  was  raised  by  the  United  States  during  World  War  I 
by- 

1 taxation  2 the  draft  3 increased  business 
•*4  getting  people  to  buy  Liberty  Eonds 

5 selling  public  lands 


Revised  35  94  46  .52 

The  United  States  government  raised  large  sums  of  money  for 
carrying  on  World  War  I by  the  sale  of 

1.  draft  numbers. 

*2  . Liberty  Bonds . 

3.  public  lands. 

4.  Liberty  ships. 

5.  commemorative  stamps. 


Revision . 1.  The  premise  has  been  reworded  to  include  ideas 

originally  in  the  distracters. 

2.  Distracters  have  been  made  parallel  in  form 
and  meaning. 
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Item  No.  32 


Original 

Federal  refers  to  the- 

1 city  2 county  3 state 


N Pu  Pi  0 

35  97  29  .71 

*4  whole  country  5 ward 


Revised  35  86  26  .61 

An  act  of  the  Federal  government  applies  to  the 

1.  city  only. 

2.  county  only. 

3.  state  only. 

*•4 . whole  country. 

5.  ward  only. 

Revision . The  item  has  been  made  a more  direct  measure  of 
~ ' attainment  by  calling  for  an  application  of 

information  rather  than  for  the  information  itself. 
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Item  No.  33 
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The  first  kind  of  writing  paper  was  called- 


1 manuscript 


-*2  papyrus  3 note  4 hand-made 
5 parchment 


Revised  35  100  57  .53 

The  writing  paper  used  by  the  ancient  Egyptians  was  called 
1.  manuscript. 

*2 . papyrus  . 

3.  note. 

4 . hand-made . 

5.  parchment. 


Revision 


The  item  has  been  made  a more  direct  measure  of 
attainment  by  substituting:  for  "first"  the  time 
at  which  the  first  paper  was  used. 


•V  .1  me*  I 


- >J  o » crtj  on  1 h 


. 


CtrffiOBW/ujm  .1 

. exrcvqaq  • $* 
. «*or  . 

~ . 


Item  No.  34 
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A person  should  be  quarantined  who- 

1 is  a member  of  a union  2 is  illiterate 

*3  has  a contaerious  disease  4 is  not  a citizen 

5 has  a criminal  record 


Revised  35  100  83  .32 

A person  should  be  kept  apart  from  others  who 

1.  is  a member  of  a union. 

2.  is  illiterate. 

*3.  has  a contagious  disease. 

4.  is  not  a citizen. 

5.  has  a criminal  record. 

Revision . The  item  has  been  made  e more  direct  measure  of 
attainment  by  substituting;  for  the  term 
"quarantined"  its  definition. 
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Item  No.  35 

N 

Pu 

Pi 

0 

Original 

35 

29 

14 

.18 

As  a result  of  the  war  with  Spain- 

*1  we  became  a world  power  2 we  lost  the  friendship  of 
Mexico  3 we  made  enemies  in  Europe  4 our  navy 
improved  5 we  received  the  Virgin  Islands 

Revised  35  23  29  -.07 

As  a result  of  the  war  with  Spain,  the  United  States 
*1 . became  a world  power. 

2.  lost  Mexican  friendship. 

3.  made  enemies  in  Europe. 

4.  improved  its  navy. 

5.  gained  the  Virgin  Islands. 

Revision . 1.  The  premise  has  been  reworded  to  include  phrases 

from  the  distracters. 

2.  Distracters  have  been  made  parallel  in  form 
and  meaning. 

Note:  The  chancre  in  emphasis  in  the  fourth  choice 
resulted  in  negative  validity  for  the  item. 
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Item  No. 

36 

N 

Pu  PI 

0 

Original 

35 

54  20 

.35 

Electricity  has  been  used 
about- 

widely  for 

lighting 

homes  for 

1 90 

years  #2  50  years  3 100 

5 500  years 

years 

4 150  years 

Revised 

35 

71  26 

.45 

Electricity  has  been  used  widely  for  lighting  homes  for 
about 

1.  90  years. 

*2.  50  years. 

3 . 100  years . 

4 . 150  years . 

5 . 200  years  . 


Revision 


No  change 
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Item  No.  37 


N Pu  Pi  <t> 


Original 


35  37  3 .42 


The  first  American  war  in  which  the  telephone  was  used 
was  the- 

1 War  of  1812  2 Civil  War  *3  Spanish- American  War 

4 World  War  I 5 The  Mexican  War 


The  first  American  War  in  which  the  telephone  could  have 
been  used  by  the  armed  forces  was  the 

1.  War  of  1812. 

2.  Civil  War. 

*3.  Spanish- American  War. 

4.  World  War  I. 

5.  Mexican  War. 


Revised 


35  20  3 .29 


Revision 


The  statement  in  the  premise  has  been  more 
precisely  worded. 
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Item  No.  38 


n pu  pi  i 


Original 


35  40  9 .36 


Manuf acturing  in  the  Middle  Ages  was  done  chiefly- 

■*1  by  guilds  2 in  cities  3 in  monasteries 
4 in  castles  5 by  labor  unions 


Manufacturing  in  the  Middle  Ages  was  done  chiefly  by 
*1 . guilds. 

2 . monks . 

3.  serfs. 

4.  labor  unions. 

5.  slaves. 

Revision.  No  chanere. 


Revised 


35  31  3 .37 
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Item  No.  39 


Ori  ginal 


N Pu  Pi  0 

35  86  34  .52 


The  invention  of  the  printing  press  was  important  chiefly 
because- 


1 it  s;ave  work  to  printers  *2  it  helped  spread  knowledge 

3 most  people  learned  to  read  when  books  were  printed 

4 monks  didn’t  have  to  copy  books  by  hand 

5  it  made  libraries  possible 


Revised 


35  71  11  .61 


The  invention  of  the  printing;  press  was  important  chiefly 
because  it  resulted  in 


1.  work  for  the  printers. 

*2.  an  easier  spread  of  knowledge. 


3.  reeding  ability  for  most  people  immediately. 

4.  no  more  copying  of  books  by  hand. 

5.  the  growth  of  large  libraries. 


Revision 


The  premise  has  been  reworded  to  include  phrases 
from  the  distrecters. 
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Item  No.  40 

N 

Pu 

Pi 

</> 

Original 

35 

40 

14 

.29 

A famous  battle  of  World  War 

I 

was  fought 

at- 

1 Manila  *2  the  Marne 

3 

London 

4 

Berlin 

5 Dunkirk 

Revised  35  57  6 ,55 

The  Grerman  advance  in  World  War  I was  stopped  at 
1.  Manila. 

#2.  the  Marne. 

3.  London. 

4.  Berlin. 

5.  Dunkirk. 

Revision. 


The  item  has  been  made  a more  direct  measure  of 
attainment  by  substituting  for  "a  famous  battle" 
the  results  of  that  battle. 
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Item  No.  41 

N 

Pu  Pi 

0 

Original 

35 

60  11 

.59 

A famous  Hebrew  who  held 
Egyptian  government  was- 

an  important 

position 

in  the 

1 Moses  #2  Joseph 

3 Aaron  4 

Joshua 

5 Elijah 

Revised 

35 

66  29 

.37 

famous  Hebrew  who  held 

an  Important 

position 

in  the 

Egyptian  government  was 
1 . Moses. 

•*2  . Jos eph . 

3.  Aaron. 

4.  Joshua. 

5.  Elijah. 


Revision . No  change. 

Note:  This  item  offered  one  of  the  few 

opportunities,  because  of  the  brevity  of  the 
distracters,  to  arrange  choices  horizontally 
in  the  original  item.  The  difference  in 
validity  in  the  two  forms  of  this  item  is  in 
keeping  with  the  findings  described  in 
Chapter  IV. 
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Item  No.  42 

N 

Pu 

Pi 

0 

Original 

35 

60 

29 

.31 

A strike  is- 

-*1  the  temporary  refusal  of  laborers  to  work  for  an  employer 
2 a boycott  3 a lockout  4 a demand  for  higher  wages 
5 a court  order  to  stop  work 

Revised  35  92  89  .05 

The  temporary  refusal  of  laborers  to  work  for  an  employer 
is  called 

*1.  a strike. 

2.  a boycott. 

3.  a lockout. 

4.  a referendum. 

5.  an  injunction. 

Revision.  The  item  has  been  reworded  in  order  to  simplify 
the  distracters  and  make  them  parallel  in  form 
and  meaning. 
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Item  No.  43 


N Pu  Pl  0 


Original 


35  63  20  .44 


The  direct  primary  is  mainly  for  the  purpose  of- 

1 deciding:  ahead  of  time  who  will  be  elected 
*2  selecting  the  candidates  who  will  run  for  office 

3 giving  the  party  bosses  a chance  to  elect  their  men 

4 putting:  out  unfit  candidates 

5 electing  officers  of  the  local  government 


It  is  possible  for  the  voters  to  select  candidates  who  will 
run  for  office  in  the 

1.  referendum. 

2.  initiative. 

3.  recall. 

*4.  direct  primary. 

5.  injunction. 


Revised 


35  69  23  .43 


Revision 


The  item  has  been  reworded  in  order  to  simplify 
the  distracters  and  to  make  them  parallel  in  form 
and  meaning. 
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Item  No*  44 


Original 


N 


0 

41 


ru 
35  54 


In  World  War  I first  use  was  made  of- 


*1  poisonous  gas  2 the  telephone  3 
4 steel  warships  5 radar 


Revised  35  71 

In  World  War  I the  first  use  was  made  of 
*1 . poisonous  gas. 

2.  the  telephone. 

3.  balloons. 

4.  steel  warships. 

5.  radar. 


Pi 

14  . 

balloons 


14  .58 


Revision 


No  change 
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Item  No.  45 

N 

Pu  Pi  0 

Original 

35 

51  6 .49 

A man  famous 

for  stamping 

out  yellow 

fever  in  Cuba  was- 

1 Gorgas 

2 Goethals 

3 Pasteur 

*4  Reed  5 Roosevelt 

Revised 

35 

77  9 .68 

man  famous  for  stamping 

out  yellow 

fever  in  Cuba  was 

1.  Cushing. 

2.  Goethals. 

3.  Pasteur. 
*4.  Reed. 

5.  Roosevelt. 


Revision . Choice  No.  1 was  considered  as  possibly  too 
close  to  correct. 
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Item  No.  46 


Original 


N 

35 


The  Christian  relie-ion  bea:an- 
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*1  before  the  Roman  Emoire  ended  2 while  Greece  ruled 
the  world  3 before  Egyptian  civilization  4 after  the 
end  of  the  Roman  Empire  5 during  the  time  of  the  Fharoahs 


Revised 


35  49  11  .41 


The  Christian  era  beeran 
*1 . before  the  Roman  Empire  ended. 

2.  while  Greece  ruled  the  world. 

3.  before  Egyptian  civilization  had  developed. 

4.  after  the  end  of  the  Roman  Empire. 

5.  during  the  time  of  the  Fharoahs. 


Revision . The  statement  in  the  premise  has  been  made  more 
nrecise . 
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Item  No.  47 

N 

Pu 

Pi 

0 

Original 

35 

69 

54 

.35 

The  Crusades  were  started  because  of- 

*1  the  capture  of  Palestine  by  the  Turks 
2 the  need  of  colonies  3 the  fall  of  Rome 
4 Mohammedanism  5 the  capture  of  Greece 

Revised  35  49  14  .38 

Military  expeditions  were  organized  in  Europe  in  the 
eleventh,  twelfth,  and  thirteenth  centuries  because  of 

*1,  the  Turkish  capture  of  Palestine. 

2.  the  need  of  colonies. 

3.  the  fall  of  the  Roman  Empire. 

4.  the  growth  of  Mohammedanism. 

5.  the  capture  of  Greece. 

Revision . 1.  The  item  has  been  made  a more  direct  measure  of 

attainment  by  substituting  a description  of  the 
,,Crusades,,  for  the  term  itself. 

2.  Distracters  have  been  made  parallel  in  form. 
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Item  No.  48 


N Pu  Pi  0 


Original 


35  69  20  .49 


The  poor  condition  of  the  South  at  the  end  of  the  Civil  War 
was  due  chiefly  to  the- 


The  hard  times  in  the  South  at  the  end  of  the  Civil  War 
were  due  chiefly  to  the 

1.  decrease  in  manufactures. 

2.  freeing  of  the  slaves. 

*3.  widespread  destruction . of  property. 

4.  difficulties  in  joining  the  Union. 

5.  small  demand  for  cotton. 


1 decrease  in  manufactures  2 freeing  of  the  slaves 
*3  destruction  of  much  property  4 difficulties  in 
joining  the  Union  5 small  demand  for  cotton 


Revised 


35  69  14  .56 


Revision 


The  statement  in  the  premise  has  been  reworded 
more  precisely. 
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Item  No.  49 

N 

Pu 

Pi 

Original 

35 

69 

17 

.53 

The  Norsemen 

did  not  affect 

the  development 

of 

America 

because  they- 

1 came  from  Iceland  *2  made  no  further  explorations 
3 were  pirates  4 did  not  try  to  find  a route  to 
the  Indies  5 were  defeated  by  the  Indians 


Revised  35  66  23  .43 

The  Norsemen  did  not  affect  the  development  of  America 
because  they  failed  to 

1.  record  their  voyages. 

*2.  make  further  explorations. 

3.  establish  trading  posts. 

4.  seek  routes  to  the  Indies. 

5.  defend  themselves  against  the  Indians. 


Revision.  1.  The  statement  in  the  premise  has  been  reworded 

to  include  phrases  from  the  distracters. 

2.  The  distracters  have  been  made  parallel  in 
form  and  meaning. 


6 > ' . O >1  rnf)^I 

>r~  r-cf:'  yd  leb  r^v  1 salt  (i  s '{J 


to  inemcol^v^b  ©flj  to  11  ron  £xb  no ’■t>e*ic M erfT 

. " ? . cv  " ' =5  ■ .*ioos»*i  . 

• 1.  0 1 ' v * • .rrl.  « •'  • ' .i:* 

. h-  c •’  >•  ‘ eiXd  • • . ' 

. ' i • I .-•  • • ■ ■ in’  • ' i?.y fens’!  • . • 


. 8fse^oa*xi 

. ni.  d:  *to1 


. 


Item  No.  50 

N 

Pu 

Pi 

Original 

35 

92 

43 

.52 

The  chief  purpose  of  a city-planning  commission  is  to- 

1 keep  factories  out  of  the  city  *2  provide  for  the 
future  growth  of  a city  3 decide  where  apartment 
houses  shall  be  built  4 dispose  of  garbage 
5 arrange  political  districts 


Revised  35  80  40  .41 

The  chief  purpose  of  a city-planning  commission  is  to 
provide  plans  for 

1.  public  buildings. 

*2.  future  growth  of  the  city. 

3.  all  public  celebrations. 

4.  regular  elections. 

5.  Community  Fund  drives. 

Revision . 1.  The  statement  in  the  premise  has  been  reworded 

to  include  phrases  from  the  distracters. 

2.  The  distracters  have  been  made  parallel  in 
form  and  meaning. 
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Item  No.  51  N Pu  0 

Original  35  74  34  .40 

An  injunction  is- 

1 for  the  purpose  of  stopping  strikes  2 a moratorium 
3 a boycott  *4  a court  order  forbidding  someone  to  do 
something  5 a strike 


Revised  35  83  23  .60 

An  order  from  the  court  forbidding  someone  to  do  something  is 

1.  a lockout. 

2.  a moratorium. 

3.  a boycott. 

*4.  an  injunction. 

5.  a strike. 

Revision . The  item  has  been  reworded  in  order  to  simplify 
the  distracters  and  to  make  them  parallel  in 
form  and  meaning. 
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Item  No.  52  N Pu 

Original  35  77 

When  ancient  Greek  civilization  existed- 

1 England  was  a civilized  country  2 Egypt  had  ceased  to 
exist  *3  most  of  Europe  was  uncivilized  4 America  had 
been  discovered  5 the  Roman  Empire  had  fallen 


Revised  35  69  20  .49 

At  the  height  of  ancient  Greek  civilization,  the  people 
were  for  the  most  part  highly  civilized  in 

1.  England. 


Pi  0 

11  .67 


2.  most  of  Europe. 
*3.  Egypt. 

4 . J ap  an . 

5.  America. 


Revision 


The  item  has  been  reworded  in  order  to  simplify 
the distracters  and  to  make  them  parallel  in  form 
and  meaning. 
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Item  No.  53 


N Pu  Pi  0 


Original 


35  31  20  .13 


Peary  reached  the  North  Pole  bv- 

1 ice  boat  2 airplane  *3  dog  sled  4 dirigible 

5 seaplane 


The  discoverer  of  the  North  Pole  made  his  discovery 
by  means  of 

1.  ice  boat. 

2.  airplane. 

*3.  dog  sled. 

4.  dirigible. 

5.  seaplane. 


Revision.  The  item  has  been  made  a more  direct  measure 
of  attainment  by  substituting  the  "deed"  for 
’’the  doer". 


Revised 


35  23  23  .00 
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Item.  No.  54  N Pu  0 

Original  35  89  74  .19 

Information  to  help  the  schools  of  the  state  is  provided 
by  the- 

1 Secretary  of  State  2 Governor  3 Attorney  General 
*4  Commissioner  (or  State  Superintendent)  of  Education 
. 5 Comptroller 

Revised  35  86  80  .08 

Information  to  help  the  schools  of  the  state  is  provided 
by  the 

1.  Secretary  of  State. 

2.  Governor. 

3.  Attorney-General. 

*4.  Commissioner  of  Education. 

5.  Comptroller. 

Revision.  No  change. 
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Item  No.  55 


N 


Pu  Pi  <t 

Original  35  57  63  -.06 

The  official  of  the  state  government  who  keeps  records  of 
state  affairs  is  the- 

1 Comini ssioner  (or  State  Superintendent)  of  Education 

2 State  Historian  #3  Secretary  of  State 

4 Comptroller  5 Clerk  of  Courts 


Revised  35  51  51  .00 

Records  of  state  affairs  are  kept  by  the 

1.  Commissioner  of  Education. 

2.  State  Historian. 

*3.  Secretary  of  State. 

4.  Comptroller. 

5.  Clerk  of  Courts. 

Revision . The  statement  in  the  premise  has  been  simplified. 
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Item  No.  56  N Pu  Pi 

Original  35  77  49 

In  a county  government,  local  county  funds  are 
by  the- 

1 Assessor  2 County  Clerk  3 Sheriff 
*4  County  Treasurer  5 County  Commissioner 


Revised  35  69  49 

In  a county  government,  local  county  funds  are 
by  the 

Al.  Assessor. 

2.  County  Clerk. 

3.  Sheriff. 

*4.  Countv  Treasurer. 

5.  County  Commissioner. 

Revision.  No  change. 
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Item  No.  57  N Pu  Pj^  0 

Original  35  46  6 .45 

The  first  telegraph  line  was  made  possible  by  the  aid  of- 

*1  Congress  2 Morse’s  own  money  3 a few  wealthy  men 
4 the  French  government  5 a tax  on  messages 

Revised  35  57  20  .38 

Samuel  F.  B.  Morse  constructed  the  first  telegraph  line 
with  the  financial  aid  of 

*1.  Congress. 

2.  no  outside  source. 

3.  a few  wealthy  men. 

4.  the  French  government. 

5.  a tax  on  messages. 

Revision.  The  premise  has  been  restated  more  precisely. 
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Item  No.  53 
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Pi 
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Original  55  23 

The  Panama  Canal  was  completed  during  the  presidency  of- 
1 Roosevelt  2 Taft  *3  Wilson  4 Coolidge  5 McKinley 


Revised  35  34  20  .16 

A direct  water  route  from  the  Grulf  of  Mexico  to  the  Pacific 
Ocean  was  made  available  for  ocean-going  ships  during  the 
presidency  of 

1#  Roosevelt. 

2.  Taft. 

#3.  Wilson. 

4.  Coolidge. 

5.  McKinley. 

Revision . 


The  item  has  been  made  a more  direct  measure  of 
attainment  by  substituting  a description  of  the 
Panama  Canal  for  the  term  itself. 
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Item  No.  59 

N 

Pu 

Pi 

ft 

Original 

35 

37 

26 

.12 

The  oath  of  office  taken  by 
President- 

Washington 

when 

he 

became 

1 was  written  by  Washington  #2  is  in  the  Constitution 

3 was  made  up  by  Congress  4 was  written  by  Jefferson 

5 was  taken  from  the  Bible 

Revised  35  37  26  .12 

The  oath  of  office  taken  by  the  President  of  the  United 
States  at  his  inauguration  may  be  found  in 

1.  Washington ' s Inaugural  Address. 

*2.  the  Constitution. 

3.  the  Congressional  Record. 

4.  Jefferson’s  diary. 

5.  the  Bible. 


Revision 


The  item  has  been  reworded  in  order  to  make 
distracters  parallel  in  form  and  meaning. 
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Item  No.  60 


Original 


N 

55 


The  Webster-Hayne  debate  was  about- 
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*1  the  powers  of  the  states  2 slavery  3 war  with 
England  4 the  tariff  5 the  growth  of  the  West 


Revised 


35 


14 


23  -.12 


The  Webster-Hayne  debate  was  concerned  with  the  same 
problem  which  caused  the 


■h-1^.  Virginia  and  Kentucky  Resolutions. 

2.  Lincoln-Douglas  debate. 

3.  Comnromise  of  1820. 

4.  Compromise  of  1850. 


5.  Kansas-Nebraska  Act. 


Revision.  The  item  has  been  made  a more  direct  measure  of 
attainment  by  requiring  the  identification  of 
two  similar  events  rather  than  the  identification 
of  a single  event. 
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